Learning to Charge More: How Q-Learning Firms Lock In Supracompetitive Prices
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Learning to Charge More".
Warren: Here's the gist of the paper. There is growing experimental evidence that Q-learning agents may learn to charge supracompetitive prices in infinite repeated games This work provides the first theoretical explanation for this behavior in infinite repeated games where firms update their pricing policies based solely on observed profits without computing equilibrium strategies Theoretical Framework and Model Setup The paper introduces a framework based on stochastic games with bounded memory to analyze the learning process The model features n firms competing over an infinite time horizon where firms choose prices based on simple one-memory policies that depend only on the current state and previous prices The analysis extends classical fixed-point theory to establish the existence of one-memory subgame perfect equilibria (SPEs) in this setting This framework allows for the analysis of dynamic collusion supported by policies like naive collusion, grim trigger policies, or increasing strategies Learning Dynamics via Q-Learning The authors study a variant of Q-learning with bounded experimentation to model how firms adapt their pricing policies based on observed outcomes They show that when the game admits both a one-stage Nash equilibrium price and a collusive-enabling price, and when the Q-function satisfies certain inequalities at the end of experimentation, firms learn to consistently charge supracompetitive prices This learning process is formalized through Algorithm 3, which combines softmax-based Q-learning with bounded experimentation Sustaining Collusion through Policy Profiles The research identifies three types of policy profiles that can support the learned supracompetitive behavior These include: 1.
Ruth: Naive collusion, where firms always choose the collusive-enabling price 2.
Warren: Grim trigger policies, which involve credible threats and dynamic escalation patterns 3.
Title and authors: Ruth: Let's talk about who wrote this for a minute. The paper is "Learning to Charge More: How Q-Learning Firms Lock In Supracompetitive Prices," and it’s authored by Cristian Chica, Yinglong Guo, and Gilad Lerman.
Warren: It’s interesting seeing Q-learning agents—which are basically reinforcement learning tools—used here to explain collusion without any explicit communication between the firms. That points to a real risk in how we think about algorithmic pricing today.
Iris: From an econometrician's view, the structure of the problem is key here. They're using a stochastic games framework with bounded memory to analyze these firms making decisions over an infinite time horizon, which means they have to consider long-term consequences.
Ruth: And that’s where the setup gets interesting for us. The model features n firms competing forever, and each firm chooses its price based only on the current state of the environment and the prices from that previous period. That's a very simple rule for an agent to follow.
Warren: A simple rule is good because it mirrors how these systems actually operate in practice, even if the agents are learning complex things through experimentation. So, they’re not assuming perfect foresight or complex optimization at every step of the decision process.
Iris: They do introduce value functions that track the expected discounted profit over time to evaluate this behavior across those infinite periods. It’s a standard tool in dynamic analysis, but here it's being used to prove something about learning patterns.
Ruth: The authors are setting up a way to rigorously test whether this learned behavior is actually stable or if it's just noise from the experimental setup itself. That’s a crucial step for any theory connecting simulation to reality.
The paper's summary: Warren: So, what’s the main takeaway from their summary of "Learning to Charge More: How Q-Learning Firms Lock In Supracompetitive Prices"? It boils down to showing that under specific conditions, these learning agents will settle into charging prices higher than what competition would naturally dictate.
Ruth: They explain that this happens when the game structure allows for both a one-stage Nash equilibrium price and a price level that enables collusion, and crucially, when the Q-function satisfies certain inequalities at the end of their experimentation phase.
Iris: The key contribution here is introducing this new class of subgame perfect equilibria they call one-memory SPEs. This gives us a formal way to describe these dynamic pricing patterns beyond just saying "they cheat."
Warren: And they show that this learned collusive behavior can be supported by three distinct types of policy profiles: naive collusion, grim trigger policies, and increasing policies. That’s a lot of structure for what seems like just an AI learning to price things.
Ruth: It’s not just one simple outcome; the paper shows that firms can learn different kinds of collusive strategies depending on how they are programmed or allowed to experiment.
Iris: They point out that naive collusion does not constitute an SPE unless the collusive-enabling price is also a one-stage Nash equilibrium, which is a really important technical distinction for verification purposes.
Warren: That means if you just assume they pick the collusive price blindly, it might not be a stable outcome in this mathematical model. It’s about the conditions being met for that behavior to stick over time.
The paper's improvements: Ruth: Now let's look at what the authors suggest as potential improvements or extensions to this work. They are focusing on making the analysis more robust and verifying these learned behaviors, not just showing they exist in theory.
Iris: One improvement they suggest is introducing a method to verify whether a given strategy profile actually constitutes a subgame perfect equilibrium using Algorithm one which involves solving linear systems derived from the value functions and checking fixed points of an operator <ref:2505.22909#pg2>.
Warren: That sounds like it moves the work from pure existence proof into something we can actually test algorithmically. It gives us a way to check if a proposed collusive pattern is mathematically sound under their model assumptions.
Ruth: Another point they raise is how to implement grim trigger strategies as one-memory subgame perfect equilibria, and they suggest doing this under specific assumptions related to the bounded memory framework of stochastic games with bounded memory.
Iris: They also address learning more sophisticated collusion patterns by utilizing Q-learning with bounded experimentation, specifically identifying sufficient conditions for increasing price behavior towards that collusive-enabling price.
Warren: So they’re moving beyond just saying "it happens" to giving us a roadmap on how to detect or build these specific types of dynamic pricing patterns in the real world using these learning mechanisms.
Ruth: They also highlight the distinction between naive collusion and grim trigger policies, noting that grim trigger policies can be supported by this framework, whereas naive collusion cannot be an SPE unless that initial collusive price is also a Nash equilibrium.
Conclusion: Warren: So to wrap up on "Learning to Charge More: How Q-Learning Firms Lock In Supracompetitive Prices," the paper shows that AI agents can indeed learn supracompetitive prices in infinite repeated games when the setup is right.
Iris: The main implication is that we need better tools—like these verification algorithms—to test if observed pricing patterns are genuine equilibrium behavior or just temporary fluctuations from their learning process.
Ruth: The paper clarifies that while naive collusion isn't always stable, strategies like grim trigger policies can be supported by this framework, which is a significant detail for policy discussions.
Warren: It’s a warning about the potential for algorithmic collusion even without any direct communication between the firms involved in the pricing decisions.
Iris: From an econometric standpoint, it reinforces how important it is to distinguish between correlation and actual equilibrium when we look at dynamic decision-making models like this one.
Ruth: So, these findings suggest that future research should focus on how to detect these specific learning mechanisms rather than just confirming they are theoretically possible. That’s where the next phase of study will likely go.
Cristian Chica, Yinglong Guo, Gilad Lerman
University of Minnesota
econ.GN, cs.AI, cs.GT, q-fin.EC
Submitted: 2025-05-28
Updated: 2026-11-01
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: There is growing experimental evidence that Q-learning agents may learn to charge supracompetitive prices in infinite repeated games This work provides the first theoretical explanation for this…
Key concepts
- Q-learning with bounded experimentation
- This is a method where firms learn optimal pricing strategies by trying different prices and observing the resulting profits. The 'bounded' aspect means they only explore a limited set of actions or prices during their learning process, helping them find stable patterns.
- One-memory policies
- Firms use simple pricing rules that depend only on the current situation and the price they set in the immediately preceding period. This limits their decision-making complexity, making it easier to analyze how they adapt their behavior over time.
- Subgame perfect equilibria (SPEs)
- An SPE is a set of strategies where every firm's choice at every possible point in the game is optimal, given what other firms are doing. The paper uses this concept to verify if learned strategies, like collusion or trigger policies, are mathematically sound.
- Collusive-enabling price
- This is a specific price level that allows firms to maintain a higher profit outcome together than they would in a purely competitive setting. Learning to charge this price indicates the emergence of cooperation or collusion.
Terminology
Summary
There is growing experimental evidence that Q-learning agents may learn to charge supracompetitive prices in infinite repeated games This work provides the first theoretical explanation for this behavior in infinite repeated games where firms update their pricing policies based solely on observed profits without computing equilibrium strategies
Theoretical Framework and Model Setup
The paper introduces a framework based on stochastic games with bounded memory to analyze the learning process The model features n firms competing over an infinite time horizon where firms choose prices based on simple one-memory policies that depend only on the current state and previous prices The analysis extends classical fixed-point theory to establish the existence of one-memory subgame perfect equilibria (SPEs) in this setting This framework allows for the analysis of dynamic collusion supported by policies like naive collusion, grim trigger policies, or increasing strategies
Learning Dynamics via Q-Learning
The authors study a variant of Q-learning with bounded experimentation to model how firms adapt their pricing policies based on observed outcomes They show that when the game admits both a one-stage Nash equilibrium price and a collusive-enabling price, and when the Q-function satisfies certain inequalities at the end of experimentation, firms learn to consistently charge supracompetitive prices This learning process is formalized through Algorithm 3, which combines softmax-based Q-learning with bounded experimentation
Sustaining Collusion through Policy Profiles
The research identifies three types of policy profiles that can support the learned supracompetitive behavior These include:
-
Naive collusion, where firms always choose the collusive-enabling price
-
Grim trigger policies, which involve credible threats and dynamic escalation patterns
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive pricing in infinite repeated games: The improved system can learn to charge supracompetitive prices by using Q-learning with bounded experimentation to distinguish between a one-stage Nash equilibrium price and a collusive-enabling price.
-
Policy verification for algorithmic collusion: The system can verify whether a given strategy profile constitutes a subgame perfect equilibrium using Algorithm 1, which involves solving linear systems derived from the value functions V1 and checking fixed points of the operator vi,s.
-
Grim trigger strategy implementation: The system can implement grim trigger strategies as one-memory subgame perfect equilibria under specific assumptions by leveraging the mathematical framework of stochastic games with bounded memory.
-
Learning sophisticated collusion patterns: The system can learn increasing price behavior towards the collusive-enabling price by utilizing Q-learning with bounded experimentation, which identifies sufficient conditions for such increasing strategies.
-
Naive collusion detection: The system can determine if learned supracompetitive prices are merely naive collusion by checking if the collusive-enabling price is a one stage Nash equilibrium, as naive collusion
does not constitute an SPE unless the collusive-enabling price is a one-stage Nash equilibrium.
Abstract
There is growing experimental evidence that Q-learning agents can learn to charge supracompetitive prices. We ask when such behavior persists, whether it emerges, and which strategies support it. In a repeated pricing game in which firms condition on the previous period's price profile, we give conditions on the learned action values under which supracompetitive prices become an absorbing state once experimentation ends. We then show that under softmax exploration these conditions are reached with positive and explicitly bounded probability, both from a random and from an all-zero initial Q-table. In the latter case firms lock in supracompetitive prices because they never learn that undercutting pays; the same mechanism can lock in the competitive price, so the long-run outcome is selected by the exploration path. Finally, we give sufficient conditions on the learned table under which the induced strategy is naive collusion, a grim trigger strategy or an increasing-price strategy; grim trigger strategies are subgame perfect under a standard discounting condition, whereas naive collusion is generally not an equilibrium.
Sources
- Artificial Intelligence and Algorithmic Price Collusion in Two-sided Markets
- Competition and Collusion in Two-Sided Markets with an Outside Option
Related papers
- Huge If True: Learning from the Size of the Surprise
- The Rebundling of Freelance Work in the AI Age
- Dynamic investment and coordination in teams: Theory and experimental evidence
- Does Frontier AI Strengthen Cyber Defense? Evidence from Project Glasswing
- Generative AI at Work
- What is Work For? Labor, Politics, and Meaning in the Age of AI