Regret, equilibrium, and learning in games: A guided tour

This note aims to serve as an entry point to the literature on learning in games, a topic with significant theoretical appeal and a wide range of applications—from machine learning and data science to economics and beyond. Our presentation is structured around two complementary viewpoints: We first consider a single agent—the learner—engaged in a sequential decision process in an unknown, non-stationary, and possibly adversarial environment. We then examine what happens when the environment is shaped by the decisions of several interacting agents, not necessarily aware of each other’s actions or goals, and all seeking to improve their individual rewards. In this general context, we examine a family of regularized learning policies based on best-responding to the past history of play, up to a regularization penalty intended to encourage exploration and prevent over-commitment to suboptimal choices. In the single-agent setting, we present some basic regret bounds for regularized learning in adversarial multi-armed bandits; in the multi-agent setting, we describe an ergodic equilibrium convergence result for zero-sum games in the spirit of classical results on fictitious play, as well as a “folk theorem” linking strategic and dynamic notions of stability—Nash equilibria and attracting points of regularized learning, respectively. We pay special attention to the information available to the players and, through a unified analysis framework, we study both oracle- and bandit, payoff-based methods. Our goal is to provide a coherent and comprehensible—albeit, by necessity, not comprehensive—account of some recent ideas in the field, and to discuss their implications for the study of rationality.

Improvements to AI systems:

1. Adaptive Exploration-Exploitation via Regularized Best-Response

• Implement a learning policy that combines best-response with a tunable regularization penalty (e.g., entropy or KL-divergence). The AI can dynamically adjust the penalty strength based on observed reward variance, preventing over-commitment to suboptimal actions in non-stationary environments.

• Capability: The AI can operate in adversarial multi-armed bandit settings with provable regret bounds (e.g., O(sqrt T K)), while maintaining robust performance when the environment shifts abruptly.

2. Multi-Agent Equilibrium Seeking with Limited Information

• Use the ergodic convergence result for zero-sum games to design AI agents that converge to Nash equilibria without full observability of opponents’ actions. The AI only needs its own payoff history (bandit feedback) or occasional oracle queries.

• Capability: Multiple AI agents can coordinate implicitly in competitive settings (e.g., auction bidding, cybersecurity defense) and reach stable, mutually optimal strategies even when they cannot observe each other’s choices or intentions.

3. Unified Framework for Oracle vs. Payoff-Based Learning

• Build a single AI module that switches between oracle-based (full information) and bandit-based (payoff-only) modes depending on the communication bandwidth or privacy constraints. The framework provides identical convergence guarantees for both modes.

• Capability: The AI can operate in privacy-sensitive or high-latency environments (e.g., federated learning, distributed control) by using payoff-based updates, then seamlessly upgrade to oracle mode when richer feedback is available, without retraining.

4. Folk-Theorem-Inspired Stability Detection

• Leverage the link between Nash equilibria and attracting points of regularized learning to create an AI that can detect whether a given strategy profile is strategically stable (i.e., robust to unilateral deviations) by analyzing the local dynamics of its own learning process.

• Capability: The AI can proactively identify and avoid unstable equilibria in multi-agent interactions (e.g., traffic routing, resource allocation), instead converging to more robust, attracting solutions that are less sensitive to other agents’ irrational behavior.

5. Non-Stationary Regret Minimization with Regularization

• Extend the single-agent regret bounds to non-stationary environments by using a time-varying regularization term that shrinks as the AI gains confidence. This allows the AI to track drifting optimal actions (e.g., in dynamic pricing or recommendation systems).

• Capability: The AI can maintain sublinear dynamic regret in changing environments, adapting to shifts in user preferences or market conditions without explicit change-point detection.

6. Exploration-Encouraging Penalty for Sparse Feedback

• Use the regularization penalty as an intrinsic exploration bonus in sparse-reward settings (e.g., reinforcement learning with delayed rewards). The AI can set the penalty proportional to the uncertainty of its value estimates, ensuring it explores uncertain actions more frequently.

• Capability: The AI can solve tasks with extremely sparse rewards (e.g., robotic navigation, game playing) by maintaining a balance between exploiting known good actions and exploring uncertain ones, with theoretical guarantees on convergence to optimal policies.

7. Decentralized Convergence for Heterogeneous Agents

• Apply the multi-agent convergence result to design AI agents with different learning rates or regularization strengths. The ergodic convergence ensures they still reach equilibrium, even if agents have asymmetric information or update frequencies.

• Capability: A swarm of AI agents (e.g., drones, IoT devices) with varying computational or communication capabilities can converge to a coordinated strategy (e.g., coverage optimization) without a central coordinator, improving scalability and robustness.

This note aims to serve as an entry point to the literature on learning in games, a topic with significant theoretical appeal and a wide range of applications -- from machine learning and data science to economics and beyond. Our presentation is structured around two complementary viewpoints: We first consider a single agent -- the learner -- engaged in a sequential decision process in an unknown, non-stationary, and possibly adversarial environment. We then examine what happens when the environment is shaped by the decisions of several interacting agents, not necessarily aware of each other's actions or goals, and all seeking to improve their individual rewards. In this general context, we examine a family of regularized learning policies based on best-responding to the past history of play, up to a regularization penalty intended to encourage exploration and prevent over-commitment to suboptimal choices. In the single-agent setting, we present some basic regret bounds for regularized learning in adversarial multi-armed bandits; in the multi-agent setting, we describe an ergodic equilibrium convergence result for zero-sum games in the spirit of classical results on fictitious play, as well as a "folk theorem" linking strategic and dynamic notions of stability -- Nash equilibria and attracting points of regularized learning, respectively. We pay special attention to the information available to the players and, through a unified analysis framework, we study both oracle- and payoff-based (bandit) methods. Our goal is to provide a coherent and comprehensible -- albeit, by necessity, not comprehensive -- account of some recent ideas in the field, and to discuss their implications for the study of rationality.