Deep Learning of Robust Market Making under Regime-Switching Order Flow

arXiv:2609.11614 · q-fin.TR · Submitted 2026-09-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: We're looking at the paper: "Deep Learning of Robust Market Making under Regime-Switching Order Flow".

Warren: Here's the gist of the paper. Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail Classical Benchmarks and the Need for Robustness Classical market-making strategies based on stochastic control, such as the Avellaneda–Stoikov and the Guéant–Lehalle–Fernandez-Tapia (GLFT) extension, provide closed-form quoting rules but rest on assumptions that break down at realistic microstructure timescales One of them is that order flow is stationary while empirical evidence points to the existence of regimes possibly associated with algorithmic execution of metaorders Existing methods provide negative PnL in this case The problem is harder still because real order flow is neither stationary nor symmetric and the prevailing regime can shift without warning The Deep Reinforcement Learning Controller A deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book—is developed In the stationary setting, RLMM outperforms GLFT across the entire observed risk–return frontier The RLMM is more robust to flow asymmetry than GLFT but still suffers large drawdowns from inventory saturation under persistent directional imbalance The state of RLMM is augmented with two auxiliary signals—a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—to restore profitability Regime Adaptation and Fine-Tuning The paper introduces a fine-tuning pipeline to address regime fragility This pipeline involves two main steps: 1.

Ruth: Fine-tuning the stationary controller on regime-switching market-order flow using two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting quotes 2.

Warren: Introducing a scenario-bandit robust fine-tuning step that upweights low-return regime scenarios to improve performance under random-persistence and correlated-direction stress tests Performance Under Stress Tests The robustness of the Algorithm B controller is tested against three stress diagnostics 1.

Ruth: Persistence sweep: The Algorithm A controller decays monotonically under persistent directional regimes, whereas Algorithm B stays profitable throughout 2.

Paper summary: Ruth: To summarize the paper "Deep Learning of Robust Market Making under Regime-Switching Order Flow," the core thesis is that classical market making strategies fail when order flow isn't stationary, which is a reality in modern markets with algorithmic metaorders.

Warren: The authors claim that they develop a deep reinforcement learning market maker called an RLMM, specifically using a Rainbow style distributional DQN, and they find it outperforms the GLFT benchmark in the stationary setting.

Iris: But the critical part isn't just the stationary performance; it’s how they tackle regime switching. They show that by augmenting the state with a Bayesian change point filter for flow bias and a queue adjusted quote exposure imbalance, profitability is substantially restored under regime switching order flow.

Ruth: Furthermore, they introduce a scenario bandit robust fine tuning step where low return regime scenarios are upweighted to improve performance under random persistence and correlated direction stress tests.

Warren: The paper uses the event driven zero intelligence Santa Fe LOB simulator to calibrate the GLFT policy through censore. This simulator is designed to model a realistic limit order book environment, though it has some limitations that they acknowledge later on.

Iris: They also identify inventory saturation under persistent directional order flow as the main failure point for stationarily trained market making agents and GLFT.

Ruth: So, it moves beyond just finding a static optimal strategy; it builds a controller that adapts its belief about the current market regime dynamically.

Conclusion: Ruth: In conclusion, this paper on "Deep Learning of Robust Market Making under Regime-Switching Order Flow" is about developing a method to make deep learning models resilient to changing market conditions.

Warren: The central contribution is that the two stage fine tuning pipeline takes a stationary optimal RLMM and makes it profitable even when order flow switches regimes, and it tightens the tail of potential profit and loss under stress tests.

Iris: It means that for someone who only listens to the show, this work shows how you can take a strategy optimized for a steady market and make it robust against the kind of non stationary behavior we actually see in real order flow.

Ruth: The paper addresses the problem of profitability under non stationary order flow by combining Bayesian regime belief with a queue adjusted exposure signal, followed by scenario bandit reweighting.

Warren: It’s about turning a stationarily trained agent into one that stays profitable under clustered directional pressure and random persistence.

Scuola Normale Superiore di Pisa

q-fin.TR

Submitted: 2026-09-10

Updated: 2026-11-12

Code: https://github.com/felipemoret77/robust-deep-market-making

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under…

Key concepts

Regime-Switching Order Flow
This refers to situations where the nature of incoming buy or sell orders changes suddenly, indicating different market conditions. Classical models assume order flow is constant, but real markets change regimes without warning, which breaks standard trading strategies.
Deep Reinforcement Learning Market Maker (RLMM)
This is an AI agent trained using a specific type of deep learning (distributional DQN) to decide the best quotes for a market maker. It learns optimal quoting rules by interacting with a simulated order book, aiming to maximize profit over time.
Scenario-Bandit Robust Fine-Tuning
This is the second stage of the pipeline where the RL agent is trained using 'scenario bandits.' This technique specifically focuses on upweighting low-return or stressful market scenarios during training. This forces the agent to learn how to handle rare, difficult conditions better.
Bayesian Online Change-Point Filter
This is a statistical tool used to detect when the underlying pattern of directional order flow has shifted. It continuously monitors incoming data and calculates a probability that a new regime has started, allowing the RL agent to adapt its strategy in real-time.

Terminology

Summary

Deep Learning of Robust Market Making under Regime-Switching Order Flow develops a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail

Classical Benchmarks and the Need for Robustness

Classical market-making strategies based on stochastic control, such as the Avellaneda–Stoikov and the Guéant–Lehalle–Fernandez-Tapia (GLFT) extension, provide closed-form quoting rules but rest on assumptions that break down at realistic microstructure timescales One of them is that order flow is stationary while empirical evidence points to the existence of regimes possibly associated with algorithmic execution of metaorders Existing methods provide negative PnL in this case The problem is harder still because real order flow is neither stationary nor symmetric and the prevailing regime can shift without warning

The Deep Reinforcement Learning Controller

A deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book—is developed In the stationary setting, RLMM outperforms GLFT across the entire observed risk–return frontier The RLMM is more robust to flow asymmetry than GLFT but still suffers large drawdowns from inventory saturation under persistent directional imbalance The state of RLMM is augmented with two auxiliary signals—a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—to restore profitability

Regime Adaptation and Fine-Tuning

The paper introduces a fine-tuning pipeline to address regime fragility This pipeline involves two main steps:

  1. Fine-tuning the stationary controller on regime-switching market-order flow using two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting quotes

  2. Introducing a scenario-bandit robust fine-tuning step that upweights low-return regime scenarios to improve performance under random-persistence and correlated-direction stress tests

Performance Under Stress Tests

The robustness of the Algorithm B controller is tested against three stress diagnostics

  1. Persistence sweep: The Algorithm A controller decays monotonically under persistent directional regimes, whereas Algorithm B stays profitable throughout

  2. Random persistence within an episode: The scenario-bandit fine-tuning improves mean terminal PnL by about 16% compared to the Algorithm B benchmark

  3. Correlated directional regimes: The scenario-bandit adjustment improves mean terminal PnL by approximately 17% versus the fixed benchmark

Economic Intuition and Value Function Interpretation

Local sensitivity slices of the learned value function reveal the economic intuition behind Algorithm B These slices show that value is largest near zero inventory and falls sharply close to the hard caps under non-zero inventory The Bayesian filter tracks the latent directional bias while the quote-exposure diagnostic provides an instantaneous measure of asymmetry in resting quotes

Conclusion

The central contribution is a two-stage fine-tuning pipeline that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail This pipeline recovers profitability under non-stationary order flow and adds robustness by upweighting low-return regime scenarios The result is an RLMM whose performance is less sensitive to random regime persistence and more resilient to clustered directional pressure

Limitations

Some limitations qualify these results, including reliance on a zero-intelligence simulator rather than a more realistic LOB simulator, the restriction of quotes to unit lots with at most one resting order per side, and the use of fixed regime width and exponential durations in the model The paper suggests future work should move toward more realistic market models and enrich the action space with order sizing and multi-level quoting

Outlook

A first direction is to move beyond the zero-intelligence simulator toward more realistic market models, and a second is to enrich the action space with order sizing and multi-level quoting and add execution-cost and queue-position features that the current compact state omits The central contribution is a two-stage fine-tuning pipeline—a Bayesian regime belief with a queue-adjusted exposure signal (Algorithm B), followed by scenario-bandit reweighting (Algorithm C)—that turns a stationary-optimal RLMM into a controller that remains profitable under regime-switching flow and has a tighter stressed terminal PnL tail

References

[1] Marco Avellaneda and Sasha Stoikov. High-frequency trading in a limit order book. p. 1

Where the paper says it

[12] apply deep RL to systematic trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market making, emphasizing scalability to high-dimensional inventory states.

Quantitative Finance, 8(3):217–224, 2008

[2] Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia. Dealing with the inventory risk: A solution to the market making problem. p. 0

Where the paper says it

Robust quoting must therefore contend not merely with average conditions but with the non-stationary, regime-dependent flow that dominates at microstructure timescales—a thread we develop across the paper’s staged training pipeline. Market making is naturally framed as a sequential stochastic-control problem: the market maker chooses bid and ask quotes, the market responds with random order arrivals and cancellations, and the market maker must trade off spread revenue against inventory risk.

Mathematics and Financial Economics, 7(4):477–507, 2013

[3] Olivier Guéant. Optimal market making. p. 1

Where the paper says it

approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments of optimal market making, and Cartea et al.

Applied Mathematical Finance, 24(2):112–154, 2017

[4] Álvaro Cartea, Sebastian Jaimungal, and José Penalva. Algorithmic and High-Frequency Trading. p. 1

Where the paper says it

[12] apply deep RL to systematic trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market making, emphasizing scalability to high-dimensional inventory states.

Cambridge University Press, Cambridge, UK, 2015

[5] Olivier Guéant. The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making. p. 1

Where the paper says it

approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments of optimal market making, and Cartea et al.

Chapman and Hall/CRC, Boca Raton, FL, 2016

[6] Álvaro Cartea, Sebastian Jaimungal, and Jason Ricci. Buy low, sell high: A high frequency trading perspective. p. 1

Where the paper says it

[12] apply deep RL to systematic trading more broadly. Guéant and Manziuk [13] apply deep RL to corporate-bond market making, emphasizing scalability to high-dimensional inventory states.

SIAM Journal on Financial Mathematics, 5(1):415–444, 2014

[7] Charles-Albert Lehalle and Sophie Laruelle. Market Microstructure in Practice. p. 5

Where the paper says it

For a discrete mid-price series {mt} sampled on the simulator clock and a lag h in physical simulated-time units, the signature curve is σb(h) = q E (mt+h − mt) 2 /h , (8) estimated empirically by its sample analogue over all overlapping increments of length h in the calibration data. For a pure diffusion dSt = σ dWt the curve is flat, σb(h) = σ; in practice microstructure noise (bid–ask bounce, tick discretization) inflates σb(h) at small lags, while at large lags the curve stabilizes at a plateau that reflects the genuine long-horizon diffusion scale of the mid-price.

World Scientific, Singapore, 2nd edition, 2018

[8] Eric Smith, J. Doyne Farmer, László Gillemot, and Supriya Krishnamurthy. Statistical theory of the continuous double auction. p. 2

Where the paper says it

2 Problem Setup and Benchmarks Market model. The core market simulator is a continuous-time, event-driven, zero-intelligence (ZI) LOB simulator, sometimes called Santa Fe model [8, 9].

Quantitative Finance, 3(6):481–514, 2003

[9] J. Doyne Farmer, Paolo Patelli, and Ilija I. Zovko. The predictive power of zero intelligence in financial markets. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

Proceedings of the National Academy of Sciences, 102(6):2254–2259, 2005

[10] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement learning. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’18, pages 434–442, Richland, SC, 2018

[11] Petter N. Kolm and Gordon Ritter. p. 1

Where the paper says it

[10] train tabular and deep agents on historical order-book data. Kolm and Ritter [11] survey the broader role of RL in finance and discuss market making as a canonical application, and Zhang et al.

Modern perspectives on reinforcement learning in finance. p. 0
Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

The Journal of Machine Learning in Finance, 1(1), 2020

[12] Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

The Journal of Financial Data Science, 2(2):25–40, 2020

[13] Olivier Guéant and Iuliia Manziuk. Deep reinforcement learning for market making in corporate bonds: Beating the curse of dimensionality. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

Applied Mathematical Finance, 26(5):387–452, 2019

[14] Bruno Gašperov and Zvonko Kostanjčar. Market making with signals through deep reinforcement learning. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

IEEE Access, 9:61611–61622, 2021

[15] Bruno Gašperov, Stjepan Beˇgušić, Petra Posedel Šimović, and Zvonko Kostanjčar. Reinforcement learning approaches to optimal market making. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

Mathematics, 9(21):2689, 2021

[16] Jonathan Sadighian. Deep reinforcement learning in cryptocurrency market making. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

arXiv preprint arXiv:1911. p. 0
Where the paper says it

com/felipemoret77/ robust-deep-market-making. 1 arXiv:2609.

p. 0
Where the paper says it

com/felipemoret77/ robust-deep-market-making. 1 arXiv:2609.

08647, 2019

[17] Thomas Spooner and Rahul Savani. Robust market making via adversarial reinforcement learning. p. 1

Where the paper says it

Related work. The use of deep reinforcement learning (RL) for market making has a growing literature that complements the closed-form controllers above by learning quoting policies directly from interaction.

In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’20, pages 2014–2016, Richland, SC, 2020

[18] Sumitra Ganesh, Nelson Vadori, Mengda Xu, Hua Zheng, Prashant Reddy, and Manuela Veloso. Reinforcement learning for market making in a multi-agent dealer market. p. 1

Where the paper says it

[15] study deep RL approaches for market making, and Sadighian [16] applies it to cryptocurrency market making, while Spooner and Savani [17] consider adversarial RL and worst-case robustness, and Ganesh et al. [18] study RL market making in a multi-agent dealer market.

arXiv preprint arXiv:1911.05892, 2019

[19] Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices: Financial Markets under the Microscope. p. 0

Where the paper says it

A final scenario-bandit step that reweights low-return regime scenarios further improves performance under random-persistence and correlated-direction stress. 1 Introduction Market makers are the intermediaries who keep modern electronic markets liquid: they continuously post two-sided quotes, standing ready to both buy and sell, and earn the bid–ask spread as compensation for supplying immediacy to the rest of the market.

Cambridge University Press, Cambridge, 2018

[20] Sophie Laruelle, Charles-Albert Lehalle, and Gilles Pagès. Optimal posting price of limit orders: Learning by trading. p. 1

Where the paper says it

Whether they remain useful in an order-driven central limit order book (defined below) is not obvious a priori, and assessing this is one objective of the paper. These assumptions are analytically convenient but typically go unverified at microstructure timescales [7], where flow is episodic, persistent, and regime-dependent, and where execution depends not only on the quote distance from the mid-price, but also on tick discreteness, finite queue depth, and the priority rules by which resting orders are matched [8, 9].

Mathematics and Financial Economics, 7(3):359–403, 2013

[21] Olivier Guéant and Charles-Albert Lehalle. General intensity shapes in optimal liquidation. p. 1

Where the paper says it

approximately symmetric order flow; see Cartea et al. [4] and Guéant [5] for textbook treatments of optimal market making, and Cartea et al.

Mathematical Finance, 25(3):457–495, 2015

[22] James D. Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357–384, 1989

[23] Bence Tóth, Imon Palit, Fabrizio Lillo, and J. Doyne Farmer. Why is equity order flow so persistent? p. 1

Where the paper says it

Whether they remain useful in an order-driven central limit order book (defined below) is not obvious a priori, and assessing this is one objective of the paper. These assumptions are analytically convenient but typically go unverified at microstructure timescales [7], where flow is episodic, persistent, and regime-dependent, and where execution depends not only on the quote distance from the mid-price, but also on tick discreteness, finite queue depth, and the priority rules by which resting orders are matched [8, 9].

Journal of Economic Dynamics and Control, 51:218–239, 2015

[24] Jean-Philippe Bouchaud, J. Doyne Farmer, and Fabrizio Lillo. p. 0

Where the paper says it

Deep Learning of Robust Market Making under Regime-Switching Order Flow Felipe Moret Fabrizio Lillo Scuola Normale Superiore di Pisa felipebueno. it, fabrizio.

How markets slowly digest changes in supply and demand. p. 0
Where the paper says it

Every fill nudges the market maker’s inventory away from neutral, and an intermediary who accumulates a large directional position is exposed to adverse price moves before that position can be unwound. The problem is harder still because real order flow is neither stationary nor symmetric: liquidity demand arrives in bursts, buy and sell pressure can persist for extended stretches, and the prevailing regime can shift without warning.

In Thorsten Hens and Klaus Reiner Schenk-Hoppé, editors, Handbook of Financial Markets: Dynamics and Evolution, pages 57–160. North-Holland, Amsterdam, 2009

[25] Fabrizio Lillo and J. Doyne Farmer. The long memory of the efficient market. p. 8

Where the paper says it

In the stationary experiments of Section 4, the auxiliary feature vector is switched off. In the regime-switching experiments of Section 6, the latent directional bias of the MO flow becomes a hidden state: the agent then needs an inference signal for the market’s 1The subscript c stands for “controller clock”, distinguishing it from the regime memory scale τr used later in Section 6 9

Studies in Nonlinear Dynamics & Econometrics, 8(3), 2004

[26] Ioanna-Yvonni Tsaknaki, Fabrizio Lillo, and Piero Mazzarisi. Online learning of order flow and market impact with Bayesian change-point detection methods. p. 2

Where the paper says it

• We identify inventory saturation under persistent directional order flow as the dominant failure of stationarily trained market-making agents and the GLFT benchmark. • We fine-tune the stationary-optimal controller on regime-switching market-order flow using two belief-style auxiliary signals: a Bayesian online change-point filter for the latent directional order flow bias and a queue-adjusted quote-exposure imbalance for the agent’s own resting quotes.

Quantitative Finance, 25(2):307–322, 2025

[27] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. p. 8

Where the paper says it

Two clocks therefore coexist: the underlying event clock on which the book evolves (continuous-time Markov), and the decision clock induced by the throttle on which the MM observes, acts, and learns. Between two consecutive decision times the simulator processes a random number Nt ≥ 1 of intermediate book events, indexed by n = 0, 1, .

John Wiley & Sons, New York, 1994

[28] Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

Artificial Intelligence, 112(1–2):181–211, 1999

[29] Ryan Prescott Adams and David J. C. MacKay. Bayesian online changepoint detection. p. 0

Where the paper says it

The RLMM is more robust to flow asymmetry than GLFT, but, like any stationarily trained strategy, it still suffers large drawdowns from inventory saturation under persistent directional imbalance. Augmenting the state of RLMM with two auxiliary signals—a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance—restores profitability.

arXiv preprint arXiv:0710. p. 0
Where the paper says it

com/felipemoret77/ robust-deep-market-making. 1 arXiv:2609.

3742, 2007

[30] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski et al., Human-level control through deep reinforcement learning. p. 7

Where the paper says it

3 Reinforcement Learning Formulation In this section, we cast the market-making problem as a reinforcement-learning problem and present the RL solutions we compare with the GLFT benchmark of Section 2. Although the simulator is event-driven, control is posed at the level of MM decisions rather than raw book events.

Nature, 518(7540):529–533, 2015

[31] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

MIT Press, Cambridge, MA, 2nd edition, 2018

[32] Richard S. Sutton. Learning to predict by the methods of temporal differences. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

Machine Learning, 3(1):9–44, 1988

[33] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. p. 0

Where the paper says it

Deep Learning of Robust Market Making under Regime-Switching Order Flow Felipe Moret Fabrizio Lillo Scuola Normale Superiore di Pisa felipebueno. it, fabrizio.

Machine Learning, 8(3):279–292, 1992

[34] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, pages 2094–2100, 2016

[35] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul et al., Rainbow: Combining improvements in deep reinforcement learning. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 3215–3222, 2018

[36] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 449–458, PMLR, 2017

[37] Ziyu Wang, Tom Schaul, Matteo Hessel et al., Dueling network architectures for deep reinforcement learning. p. 0

Where the paper says it

In this case, existing methods provide negative PnL. In this paper, we develop a deep reinforcement-learning market maker (RLMM)—a Rainbow-style distributional DQN (C51) which is calibrated and tested in a zero-intelligence limit order book.

In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1995–2003, PMLR, 2016

[38] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016

[39] Meire Fortunato et al., Noisy networks for exploration. In International Conference on Learning Representations, 2018

[40] Paul Fearnhead and Zhen Liu. On-line inference for multiple changepoint problems. p. 8

Where the paper says it

In the stationary experiments of Section 4, the auxiliary feature vector is switched off. In the regime-switching experiments of Section 6, the latent directional bias of the MO flow becomes a hidden state: the agent then needs an inference signal for the market’s 1The subscript c stands for “controller clock”, distinguishing it from the regime memory scale τr used later in Section 6 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

1

Improve market making agents by moving from stationary policies to regime-switching controllers using a Bayesian online change-point filter and a queue-adjusted quote-exposure imbalance signal to restore profitability under nonstationary order flow.

2

Implement a scenario bandit robust fine-tuning step that upweights low-return regime scenarios, allowing the reinforcement learning market maker to improve performance under random persistence and correlated direction stress tests.

3

Enhance the decision mechanism by using a compact action space of six elements and adding an inventory-gating layer that enforces a hard inventory bound to prevent directional accumulation from violating constraints.

4

Augment the state representation for the deep reinforcement learning agent with two auxiliary signals: a Bayesian online change-point filter over the directional flow bias and a queue-adjusted quote-exposure imbalance.

5

Develop an algorithm that generates an exogenous regime plan as a scenario, allowing the scenario bandit to adaptively reweight training data toward empirical lower tail losses.

Abstract

Classical market-making strategies based on stochastic control, such as the Avellaneda--Stoikov and the Guéant--Lehalle--Fernandez-Tapia (GLFT) extension, provide closed-form quoting approximations, but rest on assumptions that break down at realistic microstructure timescales. One of them is that order flow is stationary, while empirical evidence points to the existence of regimes, possibly associated with algorithmic execution of metaorders. In our experiments, persistent directional flow makes stationary controllers unprofitable. In this paper, we develop a deep reinforcement-learning market maker (RLMM)---a Rainbow-style distributional deep Q-network which is calibrated and tested in a zero-intelligence limit order book. We find that, in the stationary setting, RLMM outperforms GLFT at comparable levels of profit and loss (PnL) dispersion. The stationarily trained RLMM is more robust to flow asymmetry than GLFT, but still suffers inventory saturation and large drawdowns under persistent directional flow. Training a separate RLMM under regime-switching flow, with a Bayesian flow-belief signal and a queue-adjusted quote exposure signal, restores positive mean PnL in our experiments. A final scenario-bandit step prioritizes low-return scenarios and improves performance under varying regime persistence and directional correlation.

Sources

Related papers