From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models

arXiv:2507.18229 · econ.GN, cs.AI, q-fin.EC · Submitted 2025-07-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "From Individual Learning to Market Equilibrium".

Warren: Here's the gist of the paper. The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning agents This paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard RL agent learns a non-equilibrium, monopsonistic policy Additionally, we identify a parametric bias arising from the mismatch between economic discounting and RL’s treatment of intertemporal costs To address both issues, we propose a calibrated Mean-Field Reinforcement Learning framework that embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs Our iterative algorithm converges to a self-consistent fixed point where the agent’s policy aligns with the competitive equilibrium The Core Discrepancy in Simulation The paper identifies two distinct sources of bias that arise when naively translating economic models into RL settings The structural bias reflects what we term an agency-structure dilemma, where many economic models assume atomistic agents act as ‘takers’ of aggregate market conditions, but a naive single-agent RL simulation incentivizes the agent to become a ‘manipulator’ of its environment This leads an RL agent tasked with optimizing firm behavior to learn a policy that strategically manipulates market variables to its advantage, behaving effectively as a monopsonist The parametric bias stems from misaligned interpretations of costs and discounting, where economic models incorporate time preferences and opportunity costs through parameters like the interest rate r, while RL models represent time preferences via a fixed discount factor β and often interpret costs as per-period penalties Diagnosing Simulation Errors The authors diagnose the dual simulation bias problem by comparing the theoretical steady-state equilibrium with a naive RL simulation The structural bias is demonstrated by showing that the RL agent learns a policy that results in a market tightness far below the equilibrium level, raising the question why This is explained by analyzing the RL agent’s objective, which shows it will deliberately reduce v to suppress θ and lower w if it internalizes that θ depends on v The parametric bias is identified through a mismatch in cost specification, where the expected discounted cost of hiring in the economic model is (r + λ)c q(θ), but the RL simulation uses an effective cost of λc q(θ), leading to an underestimation of true economic costs The Proposed Unified Solution To correct both sources of bias, the paper proposes a unified framework called Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL) This approach addresses the structural bias by embedding the agent within a MFG formulation, ensuring that aggregate variables like θ are treated as exogenous during optimization but endogenously updated across iterations The parametric bias is corrected by introducing an effective cost parameter ceff, defined as ceff:= 1 + rλ/c, which internalizes both the job’s expected duration and the capital opportunity cost This calibration ensures that the RL agent evaluates its hiring decision in a way equivalent to the firm in the economic model Validation and Convergence The effectiveness of this framework is validated through simulation results, which demonstrate convergence toward the theoretical economic equilibrium The authors show that only when both corrections are applied does the learned agent behavior align with the competitive equilibrium, effectively simulating a “price-taking” firm The convergence of the RL-based mean field learning algorithm is theoretically justified by Theorem 1, which proves that under certain Lipschitz continuity assumptions on the policy map and environment response map, the composite mapping Ψ is a contraction mapping This guarantees that the iterates θk+1 = Ψ(θk) converge exponentially fast to a unique fixed point θ∗, corresponding to the equilibrium of the corrected system The ablation studies confirm that correcting only one bias is insufficient for equilibrium alignment This paper demonstrates that naively applying RL to economic models can produce systematically biased outcomes due to a mismatch between the agent’s optimization structure and the assumptions embedded in economic theory The authors conclude that correcting both structural and parametric biases is necessary for faithful computational modeling of equilibrium behavior This work offers a blueprint for more reliable application of RL in economic analysis by demanding careful alignment between the agent’s learning objective and the theoretical underpinnings of the economic environment Appendix A Equations for Steady State The equations for the steady state are presented, including several key conditions derived from the firm's optimality and market equilibrium Table 2 summarizes these theoretical steady state values for variables such as l, u, q, w, v, and θ Appendix B RL Simulation settings The simulation details specify the use of a Deep Deterministic Policy Gradient (DDPG) algorithm with an actor-critic architecture for the learning agent The state space is defined by ot = (lt, ut), and the action space consists of a single continuous action vt representing vacancy posting intensity Appendix C Ablation Study The ablation study shows that only structural correction leads to over-optimistic hiring, while only parametric correction causes the agent to behave as a market manipulator Figure 3 compares the learned θ across these two settings, showing convergence to the correct value only when both corrections are applied Appendix D Convergence of the RL-based Mean Field Fixed-Point Iteration Theorem 1 formalizes the convergence proof, stating that if Assumptions (A1) and (A2) hold and L = L1L2 < 1, then the composite mapping Ψ is a contraction mapping This leads to exponential convergence of the mean field sequence θk+1 = Ψ(θk) to a unique fixed point θ∗, which corresponds to the equilibrium of the corrected system References The paper cites numerous foundational and related works in economics and RL, including Smith

1999: , Pissarides

2000: , Sutton and Barto

2018: , and various papers on Mean Field Games The references listed include those for MF-RL, economic modeling, and deep reinforcement learning applications The reference list is extensive, covering topics from search-and-matching models to agent-based macroeconomic modeling This paper demonstrates that naively applying RL to economic models can produce systematically biased outcomes due to a mismatch between the agent’s optimization structure and the assumptions embedded in economic theory The authors conclude that correcting both structural and parametric biases is necessary for faithful computational modeling of equilibrium behavior This work offers a blueprint for more reliable application of RL in economic analysis by demanding careful alignment between the agent’s learning objective and the theoretical underpinnings of the economic environment Appendix A Equations for Steady State Equations for Steady State are provided, including several key conditions derived from the firm's optimality and market equilibrium Table 2 summarizes these theoretical steady state values for variables such as l, u, q, w, v, and θ References

Paper summary: Iris: So, to recap, this paper "From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models" points out two main problems with naive RL applications: the structural bias where agents become manipulators because the simulation incentivizes them that way, and the parametric bias from misaligning how we treat time preferences through discounting versus actual economic opportunity costs.

Ruth: That's right. The paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard reinforcement learning agent learns a non-equilibrium, monopsonistic policy.

Warren: And then they identify that parametric bias arising from the mismatch between economic discounting and the reinforcement learning treatment of intertemporal costs. That means the way time preference is modeled doesn't match how it works in reality.

Iris: To address both issues, they propose a calibrated Mean-Field Reinforcement Learning framework. This approach embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs.

Ruth: It’s about ensuring that when you have many agents, like in the aggregate market, the individual agent learns its role within that larger structure correctly.

Warren: They claim this unified framework addresses both sources of bias simultaneously. It resolves the structural issue by ensuring agents internalize their atomistic role in aggregate dynamics and fixes the parametric issue by calibrating costs to reflect the total intertemporal cost of vacancy creation implied by the economic model.

Iris: The authors show that when you apply this calibrated Mean-Field Reinforcement Learning framework, their iterative algorithm converges to a self-consistent fixed point where the agent’s policy aligns with the competitive equilibrium.

Ruth: That convergence is key because it means the simulation settles on a state that matches the theoretical economic prediction, even though we started with standard RL setups that were off course.

Warren: It suggests that instead of seeing biased results in these simulations, you need to apply this calibration work to make sure they model something real, like market equilibrium.

Iris: But the paper is careful about what it doesn't do. They note that as the number of agents increases, the joint state-action space expands exponentially, which leads to scalability issues that motivate this search for a more parsimonious framework.

Ruth: So they’re not just fixing one error; they’re building a whole new way to model these interactions that handles both the structural and parametric problems at once.

Conclusion: Warren: So, to wrap up this piece on "From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models," it really boils down to fixing two big errors when we use reinforcement learning for economics. One is the structural bias, where the agent acts like a manipulator instead of just a price taker.

Iris: And the other is parametric bias, which comes from misaligning how we handle time preferences and opportunity costs between our economic model and the reinforcement learning setup.

Ruth: To get to that real market equilibrium, you need both corrections applied simultaneously. The authors show that fixing only one of those biases isn't enough to get the simulation right.

Warren: They use a framework called Calibrated Mean-Field Reinforcement Learning to embed the agent in a field and calibrate its costs to match economic reality.

Iris: And they prove mathematically, using something called Theorem One, that this iterative process converges exponentially fast to that correct equilibrium point under specific mathematical conditions.

Ruth: It’s a blueprint for applying reinforcement learning reliably in economics because it shows exactly how to align the agent's learning goal with the underlying economic theory.

Warren: If you want your RL simulations to model something real, like market equilibrium, you can't just use a standard setup; you have to do this calibration work.

Iris: It highlights that for anyone building economic models with reinforcement learning, setting up the objective function and defining time preferences is absolutely critical.

Ruth: So the main thing here is making sure we move past just getting a simulation running and actually modeling what markets settle into when you use these techniques.

Nagoya University · Ulm University

econ.GN, cs.AI, q-fin.EC

Submitted: 2025-07-24

Updated: 2026-11-03

Journal ref: Conputational Economics (2026)

DOI: 10.1007/s10614-026-11464-y

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 73/100

The gist: The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning…

Key concepts

Structural Bias (Agency-Structure Dilemma)
This occurs when economic models assume agents are passive 'takers' of market conditions, but a naive single-agent RL setup incentivizes the agent to become an active 'manipulator.' The RL agent learns to strategically alter market variables for its own gain rather than following equilibrium rules.
Parametric Bias
This bias arises from misaligned interpretations of time preferences and costs. Economic models use parameters like the interest rate (r) to capture opportunity costs, whereas standard RL often uses a fixed discount factor ($eta$) and treats costs as simple per-period penalties, leading to inaccurate cost evaluations.
Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL)
This proposed framework embeds the agent within a larger macroeconomic field (MFG formulation) where aggregate variables are treated as fixed. It corrects the parametric bias by introducing an effective cost ($c_{eff}$) that internalizes both job duration and capital opportunity costs, ensuring the agent's hiring decisions match economic theory.
Contraction Mapping (Theorem 1)
This mathematical proof justifies the convergence of the learning algorithm. It shows that under certain conditions, the iterative update process for market variables ($ heta_{k+1} = ext{Ψ}( heta_k)$) is a contraction mapping, guaranteeing that it converges exponentially fast to a unique fixed point representing the true economic equilibrium.

Terminology

Summary

The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning agents<ref:2507. p. 018229#pg2>. This paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard RL agent learns a non-equilibrium, monopsonistic policy<ref:2507. p. 018229#pg4>. Additionally, we identify a parametric bias arising from the mismatch between economic discounting and RL’s treatment of intertemporal costs<ref:2507. p. 018229#pg5>. To address both issues, we propose a calibrated Mean-Field Reinforcement Learning framework that embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs<ref:2507. p. 018229#pg5>. Our iterative algorithm converges to a self-consistent fixed point where the agent’s policy aligns with the competitive equilibrium<ref:2507. p. 018229#pg5>.

Structural and Parametric Biases

The paper identifies two distinct sources of bias when translating economic models into RL settings: a structural bias rooted in the agent’s misperception of its environment, and a parametric bias stemming from misaligned interpretations of costs and discounting<ref:2507.18229#pg2>. The structural bias reflects what we term an agency-structure dilemma, where economic models assume atomistic agents act as ‘takers’ while standard RL environments are closed-loop systems where the agent directly shapes its environment<ref:2507. p. 018229#pg2>. As a result, an RL agent may learn a policy that strategically manipulates market variables to its advantage, behaving effectively as a monopsonist<ref:2507. p. 018229#pg4>. The parametric bias arises because economic models incorporate time preferences and opportunity costs through parameters like the interest rate r and job destruction rate λ, whereas RL models use a fixed discount factor β and interpret costs as per-period penalties<ref:2507. p. 018229#pg2>. This discrepancy means that the same cost parameter c has fundamentally different meanings across the two domains, understating the total economic cost of vacancy creation<ref:2507. p. 018229#pg5>.

Proposed Unified Correction Framework

To address these dual challenges, the authors propose a unified framework to correct both sources of bias<ref:2507.18229#pg5>. This framework involves two main corrections:

  1. Correcting the structural bias by reframing the firm’s decision problem as a Mean-Field Game (MFG), which captures strategic interdependence via a representative agent interacting with a self-consistently evolving mean field<ref:2507. p. 118229#pg5>. This ensures that individual agents correctly internalize their atomistic role in aggregate dynamics<ref:2507. p. 118229#pg7>.

  2. Correcting the parametric bias by calibrating the RL cost function to reflect the total intertemporal cost of vacancy creation implied by the economic model, adjusting for both opportunity cost of capital and expected job duration<ref:2507. p. 118229#pg5>. This calibration introduces an effective cost parameter ceff that internalizes both job’s expected duration and capital opportunity cost<ref:2507. p. 518229#pg7>.

Convergence and Validation

The proposed Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL) algorithm is designed to solve the problem under a fixed mean field θ(k) using the calibrated cost parameter ceff, iteratively updating the mean field until convergence<ref:2507.18229#pg5>. The convergence of this iterative process is justified by Theorem 1, which proves that the composite mapping Ψ is a contraction mapping under assumptions of Lipschitz continuity for both the policy map and the environment response map<ref:2507. p. 618229#pg9>. This ensures that the mean field sequence θk+1 = Ψ(θk) converges exponentially fast to a unique fixed point θ∗, which corresponds to the equilibrium of the corrected system<ref:2507. p. 718229#pg9>.

Simulation Results and Conclusion

Simulation results demonstrate that the fully corrected RL simulation converges to the theoretical economic equilibrium, closely aligning with the benchmark derived from economic theory<ref:2507.18229#pg10>. Ablation studies show that correcting only one of the two biases is insufficient; structural correction alone leads to an over-optimistic hiring where θ is too high, while parametric correction alone results in a market manipulator behavior with θ significantly below equilibrium<ref:2507. p. 118229#pg12>. This reinforces the thesis that simulating economic models with RL demands careful alignment between the agent’s learning objective and the theoretical underpinnings of the economic environment<ref:2507. p. 718229#pg10>. The paper concludes that by jointly applying MF-RL and calibrated cost formulation, both structural and parametric biases are resolved, leading to a simulation that replicates the competitive equilibrium<ref:2507. p. 818229#pg10>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL) System: The improved system will solve economic problems by embedding agents within a Mean Field Game formulation while calibrating the cost function using ceff:= 1 + r / λ c, which internalizes both the job’s expected duration and the capital opportunity cost.

  2. Structural Bias Correction: The system will avoid learning a manipulative policy by fixing aggregate variables like market tightness theta during RL training, ensuring that agents do not internalize the effect of its own actions on macro variables like theta, thus preserving the atomistic assumption of the economic model.

  3. Parametric Bias Correction: The system will accurately evaluate hiring decisions by modifying the reward function to rt = f(lt) − w(lt, θt)lt − ceffvt, ensuring that the agent's objective is equivalent to the firm in the economic model by accounting for foregone capital returns.

  4. Convergence Guarantee: The system will guarantee convergence to a valid equilibrium by utilizing Theorem 1, which proves that under Lipschitz continuity assumptions (A1 and A2), the composite mapping Ψ is a contraction mapping, ensuring that the iterates theta k+1 = Ψ(theta k) converge exponentially fast to theta∗.

  5. Robust Performance Verification: The system will demonstrate superior performance over naive RL implementations by showing that correcting only one of these issues is insufficient, confirming that both the structural and parametric corrections are necessary to achieve alignment with the competitive equilibrium.

Sources

Related papers