Deep Learning of Robust Market Making under Regime-Switching Order Flow
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.
Ruth: We're looking at the paper: "Deep Learning of Robust Market Making under Regime-Switching Order Flow".
Warren: Here's the gist of the paper. Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail Classical Benchmarks and the Need for Robustness Classical market-making strategies based on stochastic control, such as the Avellaneda–Stoikov and the Guéant–Lehalle–Fernandez-Tapia (GLFT) extension, provide closed-form quoting rules but rest on assumptions that break down at realistic microstructure timescales One of them is that order flow is stationary while empirical evidence points to the existence of regimes possibly associated with algorithmic execution of metaorders Existing methods provide negative PnL in this case The problem is harder still because real order flow is neither stationary nor symmetric and the prevailing regime can shift without warning The Deep Reinforcement Learning Controller A deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book—is developed In the stationary setting, RLMM outperforms GLFT across the entire observed risk–return frontier The RLMM is more robust to flow asymmetry than GLFT but still suffers large drawdowns from inventory saturation under persistent directional imbalance The state of RLMM is augmented with two auxiliary signals—a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—to restore profitability Regime Adaptation and Fine-Tuning The paper introduces a fine-tuning pipeline to address regime fragility This pipeline involves two main steps: 1.
Ruth: Fine-tuning the stationary controller on regime-switching market-order flow using two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting quotes 2.
Warren: Introducing a scenario-bandit robust fine-tuning step that upweights low-return regime scenarios to improve performance under random-persistence and correlated-direction stress tests Performance Under Stress Tests The robustness of the Algorithm B controller is tested against three stress diagnostics 1.
Ruth: Persistence sweep: The Algorithm A controller decays monotonically under persistent directional regimes, whereas Algorithm B stays profitable throughout 2.
Paper summary: Ruth: To summarize the paper "Deep Learning of Robust Market Making under Regime-Switching Order Flow," the core thesis is that classical market making strategies fail when order flow isn't stationary, which is a reality in modern markets with algorithmic metaorders.
Warren: The authors claim that they develop a deep reinforcement learning market maker called an RLMM, specifically using a Rainbow style distributional DQN, and they find it outperforms the GLFT benchmark in the stationary setting.
Iris: But the critical part isn't just the stationary performance; it’s how they tackle regime switching. They show that by augmenting the state with a Bayesian change point filter for flow bias and a queue adjusted quote exposure imbalance, profitability is substantially restored under regime switching order flow.
Ruth: Furthermore, they introduce a scenario bandit robust fine tuning step where low return regime scenarios are upweighted to improve performance under random persistence and correlated direction stress tests.
Warren: The paper uses the event driven zero intelligence Santa Fe LOB simulator to calibrate the GLFT policy through censore. This simulator is designed to model a realistic limit order book environment, though it has some limitations that they acknowledge later on.
Iris: They also identify inventory saturation under persistent directional order flow as the main failure point for stationarily trained market making agents and GLFT.
Ruth: So, it moves beyond just finding a static optimal strategy; it builds a controller that adapts its belief about the current market regime dynamically.
Conclusion: Ruth: In conclusion, this paper on "Deep Learning of Robust Market Making under Regime-Switching Order Flow" is about developing a method to make deep learning models resilient to changing market conditions.
Warren: The central contribution is that the two stage fine tuning pipeline takes a stationary optimal RLMM and makes it profitable even when order flow switches regimes, and it tightens the tail of potential profit and loss under stress tests.
Iris: It means that for someone who only listens to the show, this work shows how you can take a strategy optimized for a steady market and make it robust against the kind of non stationary behavior we actually see in real order flow.
Ruth: The paper addresses the problem of profitability under non stationary order flow by combining Bayesian regime belief with a queue adjusted exposure signal, followed by scenario bandit reweighting.
Warren: It’s about turning a stationarily trained agent into one that stays profitable under clustered directional pressure and random persistence.
Scuola Normale Superiore di Pisa
q-fin.TR
Submitted: 2026-09-10
Updated: 2026-11-12
Code: https://github.com/felipemoret77/robust-deep-market-making
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under…
Key concepts
- Regime-Switching Order Flow
- This refers to situations where the nature of incoming buy or sell orders changes suddenly, indicating different market conditions. Classical models assume order flow is constant, but real markets change regimes without warning, which breaks standard trading strategies.
- Deep Reinforcement Learning Market Maker (RLMM)
- This is an AI agent trained using a specific type of deep learning (distributional DQN) to decide the best quotes for a market maker. It learns optimal quoting rules by interacting with a simulated order book, aiming to maximize profit over time.
- Scenario-Bandit Robust Fine-Tuning
- This is the second stage of the pipeline where the RL agent is trained using 'scenario bandits.' This technique specifically focuses on upweighting low-return or stressful market scenarios during training. This forces the agent to learn how to handle rare, difficult conditions better.
- Bayesian Online Change-Point Filter
- This is a statistical tool used to detect when the underlying pattern of directional order flow has shifted. It continuously monitors incoming data and calculates a probability that a new regime has started, allowing the RL agent to adapt its strategy in real-time.
Terminology
Summary
Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail
Classical Benchmarks and the Need for Robustness
Classical market-making strategies based on stochastic control, such as the Avellaneda–Stoikov and the Guéant–Lehalle–Fernandez-Tapia (GLFT) extension, provide closed-form quoting rules but rest on assumptions that break down at realistic microstructure timescales One of them is that order flow is stationary while empirical evidence points to the existence of regimes possibly associated with algorithmic execution of metaorders Existing methods provide negative PnL in this case The problem is harder still because real order flow is neither stationary nor symmetric and the prevailing regime can shift without warning
The Deep Reinforcement Learning Controller
A deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book—is developed In the stationary setting, RLMM outperforms GLFT across the entire observed risk–return frontier The RLMM is more robust to flow asymmetry than GLFT but still suffers large drawdowns from inventory saturation under persistent directional imbalance The state of RLMM is augmented with two auxiliary signals—a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—to restore profitability
Regime Adaptation and Fine-Tuning
The paper introduces a fine-tuning pipeline to address regime fragility This pipeline involves two main steps:
-
Fine-tuning the stationary controller on regime-switching market-order flow using two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting quotes
-
Introducing a scenario-bandit robust fine-tuning step that upweights low-return regime scenarios to improve performance under random-persistence and correlated-direction stress tests
Performance Under Stress Tests
The robustness of the Algorithm B controller is tested against three stress diagnostics
-
Persistence sweep: The Algorithm A controller decays monotonically under persistent directional regimes, whereas Algorithm B stays profitable throughout
-
Random persistence within an episode: The scenario-bandit fine-tuning improves mean terminal PnL by about 16% compared to the Algorithm B benchmark
-
Correlated directional regimes: The scenario-bandit adjustment improves mean terminal PnL by approximately 17% versus the fixed benchmark
Economic Intuition and Value Function Interpretation
Local sensitivity slices of the learned value function reveal the economic intuition behind Algorithm B These slices show that value is largest near zero inventory and falls sharply close to the hard caps under non-zero inventory The Bayesian filter tracks the latent directional bias while the quote-exposure diagnostic provides an instantaneous measure of asymmetry in resting quotes
Conclusion
The central contribution is a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail This pipeline recovers profitability under non-stationary order flow and adds robustness by upweighting low-return regime scenarios The result is an RLMM whose performance is less sensitive to random regime persistence and more resilient to clustered directional pressure
Limitations
Some limitations qualify these results, including reliance on a zero-intelligence simulator rather than a more realistic LOB simulator, the restriction of quotes to unit lots with at most one resting order per side, and the use of fixed regime width and exponential durations in the model The paper suggests future work should move toward more realistic market models and enrich the action space with order sizing and multi-level quoting
Outlook
A first direction is to move beyond the zero-intelligence simulator toward more realistic market models, and a second is to enrich the action space with order sizing and multi-level quoting and add execution-cost and queue-position features that the current compact state omits The central contribution is a two-stage fine-tuning pipeline—a Bayesian regime belief with a queue-adjusted exposure signal (Algorithm B), followed by scenario-bandit reweighting (Algorithm C)—that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail
References
[1] Marco Avellaneda and Sasha Stoikov. High-frequency trading in a limit order book. p. 1 [12] apply deep RL to systematic
trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market
making, emphasizing scalability to high-dimensional inventory states.Where the paper says it
[2] Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia. Dealing with the inventory risk: A solution to the market making problem. p. 0 Robust quoting must therefore
contend not merely with average conditions but with the non-stationary, regime-dependent flow
that dominates at microstructure timescales—a thread we develop across the paper’s staged
training pipeline. Market making is naturally framed as a sequential stochastic-control problem: the market
maker chooses bid and ask quotes, the market responds with random order arrivals and cancellations, and the market maker must trade off spread revenue against inventory risk.Where the paper says it
[3] Olivier Guéant. Optimal market making. p. 1 approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments
of optimal market making, and Cartea et al.Where the paper says it
[4] Álvaro Cartea, Sebastian Jaimungal, and José Penalva. Algorithmic and High-Frequency Trading. p. 1 [12] apply deep RL to systematic
trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market
making, emphasizing scalability to high-dimensional inventory states.Where the paper says it
[5] Olivier Guéant. The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making. p. 1 approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments
of optimal market making, and Cartea et al.Where the paper says it
[6] Álvaro Cartea, Sebastian Jaimungal, and Jason Ricci. Buy low, sell high: A high frequency trading perspective. p. 1 [12] apply deep RL to systematic
trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market
making, emphasizing scalability to high-dimensional inventory states.Where the paper says it
[7] Charles-Albert Lehalle and Sophie Laruelle. Market Microstructure in Practice. p. 5 For a discrete mid-price series {mt} sampled
on the simulator clock and a lag h in physical simulated-time units, the signature curve is
σb(h) = q
E
(mt+h − mt)
2
/h , (8)
estimated empirically by its sample analogue over all overlapping increments of length h in
the calibration data. For a pure diffusion dSt = σ dWt the curve is flat, σb(h) = σ; in practice
microstructure noise (bid–ask bounce, tick discretization) inflates σb(h) at small lags, while at
large lags the curve stabilizes at a plateau that reflects the genuine long-horizon diffusion scale of
the mid-price.Where the paper says it
[8] Eric Smith, J. Doyne Farmer, László Gillemot, and Supriya Krishnamurthy. Statistical theory of the continuous double auction. p. 2 2 Problem Setup and Benchmarks
Market model. The core market simulator is a continuous-time, event-driven, zero-intelligence
(ZI) LOB simulator, sometimes called Santa Fe model [8, 9].Where the paper says it
[9] J. Doyne Farmer, Paolo Patelli, and Ilija I. Zovko. The predictive power of zero intelligence in financial markets. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[10] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement learning. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction.Where the paper says it
[11] Petter N. Kolm and Gordon Ritter. p. 1 [10] train tabular and deep agents on historical
order-book data. Kolm and Ritter [11] survey the broader role of RL in finance and discuss
market making as a canonical application, and Zhang et al. In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
Where the paper says it
[12] Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[13] Olivier Guéant and Iuliia Manziuk. Deep reinforcement learning for market making in corporate bonds: Beating the curse of dimensionality. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction.Where the paper says it
[14] Bruno Gašperov and Zvonko Kostanjčar. Market making with signals through deep reinforcement learning. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction.Where the paper says it
[15] Bruno Gašperov, Stjepan Beˇgušić, Petra Posedel Šimović, and Zvonko Kostanjčar. Reinforcement learning approaches to optimal market making. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction.Where the paper says it
[16] Jonathan Sadighian. Deep reinforcement learning in cryptocurrency market making. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction. com/felipemoret77/
robust-deep-market-making. 1
arXiv:2609. com/felipemoret77/
robust-deep-market-making. 1
arXiv:2609.Where the paper says it
Where the paper says it
Where the paper says it
[17] Thomas Spooner and Rahul Savani. Robust market making via adversarial reinforcement learning. p. 1 Related work. The use of deep reinforcement learning (RL) for market making has a growing
literature that complements the closed-form controllers above by learning quoting policies
directly from interaction.Where the paper says it
[18] Sumitra Ganesh, Nelson Vadori, Mengda Xu, Hua Zheng, Prashant Reddy, and Manuela Veloso. Reinforcement learning for market making in a multi-agent dealer market. p. 1 [15] study deep RL approaches for market making, and
Sadighian [16] applies it to cryptocurrency market making, while Spooner and Savani [17]
consider adversarial RL and worst-case robustness, and Ganesh et al. [18] study RL market
making in a multi-agent dealer market.Where the paper says it
[19] Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices: Financial Markets under the Microscope. p. 0 A final scenario-bandit step that reweights low-return regime scenarios further
improves performance under random-persistence and correlated-direction stress. 1 Introduction
Market makers are the intermediaries who keep modern electronic markets liquid: they continuously post two-sided quotes, standing ready to both buy and sell, and earn the bid–ask spread
as compensation for supplying immediacy to the rest of the market.Where the paper says it
[20] Sophie Laruelle, Charles-Albert Lehalle, and Gilles Pagès. Optimal posting price of limit orders: Learning by trading. p. 1 Whether they remain useful in an order-driven central limit order book (defined below)
is not obvious a priori, and assessing this is one objective of the paper. These assumptions are
analytically convenient but typically go unverified at microstructure timescales [7], where flow is
episodic, persistent, and regime-dependent, and where execution depends not only on the quote
distance from the mid-price, but also on tick discreteness, finite queue depth, and the priority
rules by which resting orders are matched [8, 9].Where the paper says it
[21] Olivier Guéant and Charles-Albert Lehalle. General intensity shapes in optimal liquidation. p. 1 approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments
of optimal market making, and Cartea et al.Where the paper says it
[22] James D. Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357–384, 1989
[23] Bence Tóth, Imon Palit, Fabrizio Lillo, and J. Doyne Farmer. Why is equity order flow so persistent? p. 1 Whether they remain useful in an order-driven central limit order book (defined below)
is not obvious a priori, and assessing this is one objective of the paper. These assumptions are
analytically convenient but typically go unverified at microstructure timescales [7], where flow is
episodic, persistent, and regime-dependent, and where execution depends not only on the quote
distance from the mid-price, but also on tick discreteness, finite queue depth, and the priority
rules by which resting orders are matched [8, 9].Where the paper says it
[24] Jean-Philippe Bouchaud, J. Doyne Farmer, and Fabrizio Lillo. p. 0 Deep Learning of Robust Market Making
under Regime-Switching Order Flow
Felipe Moret Fabrizio Lillo
Scuola Normale Superiore di Pisa
felipebueno. it, fabrizio. Every fill nudges the market maker’s inventory away from neutral, and an intermediary who
accumulates a large directional position is exposed to adverse price moves before that position
can be unwound. The problem is harder still because real order flow is neither stationary nor
symmetric: liquidity demand arrives in bursts, buy and sell pressure can persist for extended
stretches, and the prevailing regime can shift without warning.Where the paper says it
Where the paper says it
[25] Fabrizio Lillo and J. Doyne Farmer. The long memory of the efficient market. p. 8 In the stationary experiments of Section 4, the auxiliary feature vector is
switched off. In the regime-switching experiments of Section 6, the latent directional bias of
the MO flow becomes a hidden state: the agent then needs an inference signal for the market’s
1The subscript c stands for “controller clock”, distinguishing it from the regime memory scale τr used later in
Section 6
9Where the paper says it
[26] Ioanna-Yvonni Tsaknaki, Fabrizio Lillo, and Piero Mazzarisi. Online learning of order flow and market impact with Bayesian change-point detection methods. p. 2 • We identify inventory saturation under persistent directional order flow as the dominant
failure of stationarily trained market-making agents and the GLFT benchmark. • We fine-tune the stationary-optimal controller on regime-switching market-order flow using
two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional
order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting
quotes.Where the paper says it
[27] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. p. 8 Two clocks therefore coexist: the underlying event clock on which the book
evolves (continuous-time Markov), and the decision clock induced by the throttle on which the
MM observes, acts, and learns. Between two consecutive decision times the simulator processes
a random number Nt ≥ 1 of intermediate book events, indexed by n = 0, 1, .Where the paper says it
[28] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[29] Ryan Prescott Adams and David J. C. MacKay. Bayesian online changepoint detection. p. 0 The RLMM is
more robust to flow asymmetry than GLFT, but, like any stationarily trained strategy, it still
suffers large drawdowns from inventory saturation under persistent directional imbalance. Augmenting the state of RLMM with two auxiliary signals—a Bayesian online change-point
filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—restores
profitability. com/felipemoret77/
robust-deep-market-making. 1
arXiv:2609.Where the paper says it
Where the paper says it
[30] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski et al., Human-level control through deep reinforcement learning. p. 7 3 Reinforcement Learning Formulation
In this section, we cast the market-making problem as a reinforcement-learning problem and
present the RL solutions we compare with the GLFT benchmark of Section 2. Although the
simulator is event-driven, control is posed at the level of MM decisions rather than raw book
events.Where the paper says it
[31] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[32] Richard S. Sutton. Learning to predict by the methods of temporal differences. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[33] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. p. 0 Deep Learning of Robust Market Making
under Regime-Switching Order Flow
Felipe Moret Fabrizio Lillo
Scuola Normale Superiore di Pisa
felipebueno. it, fabrizio.Where the paper says it
[34] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[35] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul et al., Rainbow: Combining improvements in deep reinforcement learning. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[36] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[37] Ziyu Wang, Tom Schaul, Matteo Hessel et al., Dueling network architectures for deep reinforcement learning. p. 0 In this case, existing
methods provide negative PnL. In this paper, we develop a deep reinforcement-learning
market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated
and tested in a zero-intelligence limit order book.Where the paper says it
[38] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016
[39] Meire Fortunato et al., Noisy networks for exploration. In International Conference on Learning Representations, 2018
[40] Paul Fearnhead and Zhen Liu. On-line inference for multiple changepoint problems. p. 8 In the stationary experiments of Section 4, the auxiliary feature vector is
switched off. In the regime-switching experiments of Section 6, the latent directional bias of
the MO flow becomes a hidden state: the agent then needs an inference signal for the market’s
1The subscript c stands for “controller clock”, distinguishing it from the regime memory scale τr used later in
Section 6
9Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
1
Improve market making agents by moving from stationary policies to regime-switching controllers using a Bayesian online change-point filter and a queue-adjusted quote-exposure imbalance signal to restore profitability under nonstationary order flow.
2
Implement a scenario bandit robust fine-tuning step that upweights low-return regime scenarios, allowing the reinforcement learning market maker to improve performance under random persistence and correlated direction stress tests.
3
Enhance the decision mechanism by using a compact action space of six elements and adding an inventory-gating layer that enforces a hard inventory bound to prevent directional accumulation from violating constraints.
4
Augment the state representation for the deep reinforcement learning agent with two auxiliary signals: a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance.
5
Develop an algorithm that generates an exogenous regime plan as a scenario, allowing the scenario bandit to adaptively reweight training data toward empirical lower tail losses.
Abstract
Classical market-making strategies based on stochastic control, such as the Avellaneda--Stoikov and the Guéant--Lehalle--Fernandez-Tapia (GLFT) extension, provide closed-form quoting approximations, but rest on assumptions that break down at realistic microstructure timescales. One of them is that order flow is stationary, while empirical evidence points to the existence of regimes, possibly associated with algorithmic execution of metaorders. In our experiments, persistent directional flow makes stationary controllers unprofitable. In this paper, we develop a deep reinforcement-learning market maker (RLMM)---a Rainbow-style distributional deep Q-network which is calibrated and tested in a zero-intelligence limit order book. We find that, in the stationary setting, RLMM outperforms GLFT at comparable levels of profit and loss (PnL) dispersion. The stationarily trained RLMM is more robust to flow asymmetry than GLFT, but still suffers inventory saturation and large drawdowns under persistent directional flow. Training a separate RLMM under regime-switching flow, with a Bayesian flow-belief signal and a queue-adjusted quote exposure signal, restores positive mean PnL in our experiments. A final scenario-bandit step prioritizes low-return scenarios and improves performance under varying regime persistence and directional correlation.
Sources
- Deep Reinforcement Learning in Cryptocurrency Market Making
- Reinforcement Learning for Market Making in a Multi-agent Dealer Market
- Bayesian Online Changepoint Detection
Related papers
- AIDA: An AI Outage Registry with Evidence from On-Chain MEV
- MLP-Initialized Reinforcement Learning for Exit Timing in SPX Short-Straddle Portfolios
- Beyond Anonymous Order Flow: Wallet Identity as a Feature for Short-Horizon Return Prediction
- Combinatorial Arbitrage in Prediction Markets
- TradeMech: A Method to Multilaterally Net Trades Without Altering Counterparty Exposure
- Optimal Trading Speed on Blockchain