Optimal execution in a fractional setting with reinforcement learning

arXiv:2611.11373 · q-fin.TR, q-fin.CP, q-fin.ST · Submitted 2026-11-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Optimal execution in a fractional setting with reinforcement learning".

Warren: Here's the gist of the paper. Optimal liquidation in a fractional setting with reinforcement learning develops a strategy to exploit information induced by the correlation between price increments, outperforming deterministic solutions when asset dynamics follow a fractional Brownian motion This strategy is derived from a policy gradient algorithm parameterized by a neural network trained to maximize the CARA utility Problem Formulation and Motivation The optimal liquidation problem considers selling a quantity Q of shares over time T with linear market impact, but for fractional Brownian motion, the deterministic solution is no longer optimal The paper focuses on maximizing the constant absolute risk aversion (CARA) utility defined by formula (2), which allows for the application of Bellman’s principle The authors introduce a policy gradient algorithm parameterized by a neural network to numerically approximate the optimal dynamic controls under this criterion Model and Feature Engineering The model uses an implementation shortfall benchmark where the payoff is defined as XT − QS0, and the objective is to solve the control problem fined by formula (2) The features fed to the network are denoted as Ft = (t, qt, St, Rˆt,t+τ) where Rˆt,t+τ is the conditional expectation of the future price return over a horizon τ The optimal lags for the forecast are computed via numerical optimization given equation (7), which depends only on H Training Algorithm and Policy Gradient The policy gradient algorithm is designed for a time discrete problem, dividing

0, T: into N equal intervals at times (tk = kT/N) The policy is parameterized by πθ(q F) = N µθ(F), σ2explo, where µθ(F) is a multilayer perceptron The reward function used in the reinforcement learning setup is defined as Gnn = − exp (−λΠnnT), and the objective is to maximize J(θ) = Eπθ Gnn A modified gradient approximation is used to reduce variance by comparing the learned policy to the AC baseline strategy Performance and Insights The learned policy outperforms the Almgren-Chriss baseline for every H ̸= 1/2 and recovers it at H = 1/2 The gain is larger when H is far from 1/2, with an asymmetry favoring the anti-persistent regime The visualization of individual trajectories reveals two qualitatively distinct behaviors: trend-following with short positions when increments are persistent, and inventory-holding to capture mean-reversion when they are anti-persistent The method is robust to estimation error in (H, σ), as even a 20% relative error on (H, σ) only costs about 5% of the certainty equivalent The developed policy gradient algorithm learns a liquidation strategy under fBm dynamics that outperforms the AC baseline for every tested value of H ̸= 1/2 The gain grows as H moves away from 1/2 and is markedly larger in the anti-persistent regime A moderate number of lags suffices, with most of the gain captured at p = 5 The method is robust to parameter misspecification References

Title and authors: Ruth: So this paper is about finding a way to sell shares over time when you're dealing with something called fractional Brownian motion. It’s building on the classic work by Almgren and Chriss about market impact, but for this new type of asset dynamics, their deterministic solution just doesn't work anymore.

Warren: Right, so when you introduce that fractional Brownian motion—which models long-term memory in price movements—the standard mean-variance trade-off they use isn't the best choice for maximizing the payoff. They focus instead on something called CARA utility, which is a different way of looking at risk aversion.

Iris: That CARA utility is what lets them use Bellman's principle to set up an Hamilton-Jacobi-Bellman equation, which means they can actually find an optimal control strategy under these new conditions. The paper then introduces a policy gradient algorithm using a neural network to find the actual dynamic controls.

Ruth: Exactly, so it’s not just finding the best fixed speed or timing; it’s creating a strategy that changes based on what the asset is doing in real-time, using that reinforcement learning approach to maximize this utility function. It moves away from just looking at past data to actually learning how to act optimally given the complex price increments.

Warren: And they feed the neural network a specific set of features, Ft, which includes time and current position and then conditional expectations of future returns over a short horizon, tau. This is how the AI gets its picture of what’s going on right now to decide what to do next.

Iris: That feature set is key because they figure out the optimal lags for those forecasts—the best time window to look into the future—through numerical optimization based on some parameter H in their model, which dictates how persistent or anti-persistent the price movements are.

Ruth: And then they show how this strategy performs against the standard Almgren and Chriss baseline, finding that it outperforms it for every value of H except when H is exactly one half. That suggests a real advantage when you can exploit those specific correlation structures in the asset's movement.

Title and authors: Warren: The results are interesting because they find that the performance gain gets bigger as H moves further away from one half, and there’s an asymmetry favoring periods where price changes tend to persist—that is, when H is greater than one half.

Iris: I noticed something else about the behavior of this learned policy when you look at it visually; for persistent regimes where H is greater than one half, the system learns a trend-following approach with short positions when those increments are correlated.

Ruth: That makes sense from an incentive perspective; if the price keeps moving in one direction, trying to sell quickly in that direction might actually be more effective than trying to stay neutral. But then you have the opposite scenario where H is less than one half.

Warren: In that anti-persistent regime, when increments are anti-persistent, the strategy learns to hold inventory and capture mean reversion instead of just following the immediate trend. It’s a switch based on the nature of the price correlation itself.

Iris: The paper also claims that this whole setup is quite robust; they tested it even with some errors in estimating H and sigma, finding that a twenty percent relative error on those parameters only costs about five percent of the final certainty equivalent payoff.

Ruth: That robustness is important for real-world trading because you never have perfect knowledge of the underlying asset's memory or volatility, so knowing that a small mistake won't ruin your entire strategy gives it more practical value.

Warren: So to sum up this paper, they developed a policy gradient algorithm parameterized by a neural network to find optimal liquidation strategies under fractional Brownian motion dynamics by maximizing CARA utility, and they showed this method beats the Almgren and Chriss baseline for almost all scenarios.

Iris: I just want to check that identification; the work relies on using the CARA utility to derive an HJB equation, which then leads to a policy gradient algorithm where the reward function is defined as minus exp of lambda times pi times T. This setup seems well-defined for finding the optimal dynamic controls, provided you have a way to numerically approximate that policy.

Title and authors: Ruth: That numerical approximation is where the engineering comes in; they break up the time period T into N equal intervals and use a multilayer perceptron to parameterize the policy function, which helps them solve this continuous problem in a discrete setting.

Warren: And for those lags they feed into the network, they calculate them through numerical optimization based on some equation that only depends on H, which is pretty clever because it ties the look-back window directly to the asset's memory.

Iris: That dependence on H is crucial because it means that you really do need a separate model for training if you want to test different Hurst exponents, since "the entire pipeline is H-specific: the Hurst exponent must be known both to simulate the training trajectories and to compute forecasts."

Ruth: So, looking ahead, this suggests that for fractional dynamics in execution problems, we shouldn't just stick to the classical models; we need these data-driven reinforcement learning approaches conditioned on those complex correlation structures.

Warren: The practical implication is that if you are trading assets where price history matters a lot—like commodities or certain equities—you should consider modeling the asset as having long-range dependence instead of just simple Brownian motion when you design your liquidation strategy.

Iris: And for future work, they point out that while they showed robustness to parameter estimation errors, the paper doesn't explore how this framework adapts if the market impact isn't perfectly linear, which is another friction in execution problems.

Ruth: It does leave room for exploring those non-linear market impact scenarios or perhaps looking at how the learning process itself could be adapted to incorporate more complex risk constraints beyond just CARA utility.

Warren: So that's the picture with this paper on optimal execution in a fractional setting with reinforcement learning, showing how to use AI to handle memory effects in trading dynamics.

The paper's summary: Ruth: So, this paper is about how to sell shares over time when you're dealing with something called fractional Brownian motion, and they’ve used reinforcement learning to find a way that beats the standard deterministic solutions for selling stocks.

Warren: Basically, they found a strategy that uses neural networks to learn the best timing and size for selling based on how much price history matters in an asset.

Iris: The core idea is that because price movements have long memory under fractional Brownian motion, you need a dynamic plan that adjusts constantly rather than a fixed schedule.

Ruth: Exactly. They use this reinforcement learning approach to maximize something called CARA utility, which is basically their definition of risk aversion for the selling problem.

Warren: The feature set they feed into the neural network includes things like time and how much you've already sold, plus some forecasts about future price returns over a short window tau.

Iris: They figured out the best look-back windows for those forecasts by doing some numerical optimization that only depends on the parameter H, which tells you how persistent or anti-persistent the price moves are.

Ruth: And here’s where it gets interesting for execution—they found this learned policy beats the standard Almgren and Chriss baseline for every value of H except when H is exactly one half.

Warren: That means if you can figure out that correlation structure, your AI strategy gives you a real edge over the traditional methods.

Iris: The gain gets bigger as H moves further away from one half, and there’s a specific asymmetry favoring those anti-persistent price movements where the asset tends to revert.

Ruth: The qualitative results are pretty telling too. for persistent price changes, the AI learns to be trend-following with short positions when increments are correlated.

Warren: But in that anti-persistent regime, when they aren't correlated, the strategy learns to hold inventory and capture mean reversion instead of just chasing the immediate trend.

Iris: What I see is that this whole setup is pretty robust; they tested it even with some errors in estimating H and sigma, finding that a twenty percent relative error on those parameters only costs about five percent of the final certainty equivalent payoff.

Ruth: That robustness is key for real trading because you never have perfect knowledge of how much memory an asset has or its exact volatility.

Warren: So to sum up, they’ve developed a policy gradient algorithm that learns an optimal liquidation strategy under fractional Brownian motion dynamics by maximizing CARA utility, and it shows this method beats the Almgren and Chriss baseline for almost all scenarios.

Iris: They confirm that the variation doing the work is this learned policy function, parameterized by a multilayer perceptron, which helps them solve this continuous problem in a discrete setting.

Ruth: And they show that most of the performance gain comes from just five lag intervals, suggesting you don't need an impossibly long memory to get good results.

Warren: The implication is that if you’re trading assets where price history really matters—like commodities or certain equities—you should consider modeling them with long-range dependence instead of just simple Brownian motion when designing your liquidation strategy.

Iris: For future work, they point out that while it handles estimation errors well, they haven't explored what happens if the market impact isn't perfectly linear, which is another friction in execution problems.

Ruth: That leaves room to see how this framework could adapt if the cost of trading shares wasn't just a simple linear function of the quantity sold.

The paper's improvements: Tom: So, we just covered how this paper uses reinforcement learning to find better selling strategies when price history matters under fractional Brownian motion, and now we're going to look at how they suggest improving that approach.

Ruth: They aren't just happy with their initial results; they suggest a few ways to make the whole system more flexible.

Warren: They propose using different lags for those future price return forecasts, not just sticking to the ones they found through numerical optimization based on H.

Iris: This means instead of having one fixed set of look-back windows, you could have several options and let the AI choose which ones are best depending on what the market is actually doing at that moment.

Ruth: That makes sense because market conditions change, so a fixed lag might be good in one price movement pattern but totally useless in another.

Warren: From a trading cost standpoint, if the AI can dynamically adjust its look-back window based on current market signals, it should reduce the risk of making poor trade decisions during volatile transitions.

Iris: The authors also hint at exploring how this framework could handle different types of market impact more effectively than just assuming a linear cost for selling shares.

Ruth: That’s a big one because most execution models assume that if you sell a lot, the market impact scales linearly with the size of your trade, but that might not be true in complex markets.

Warren: If they can model non-linear costs better, their strategy won't just be better for fractional dynamics; it’ll be robust across different asset types too.

Iris: The improvement here would involve making the AI learn to account for those changing market friction costs alongside the price memory itself.

Ruth: That brings us to the next big question: what happens when we take this concept and apply it to something more complex than just selling shares, like managing a whole portfolio?

Warren: Or maybe what does this mean for people who only listen to the show—how does this idea of learning based on memory change how you think about risk in general?

Conclusion: Tom: So we're wrapping up on this paper, "Optimal execution in a fractional setting with reinforcement learning," which shows how AI can find better selling strategies when price history matters under fractional Brownian motion and beat the standard deterministic solutions.

Ruth: It really boils down to using reinforcement learning to handle the uncertainty introduced by long-term memory in asset prices, maximizing that CARA utility function we talked about earlier.

Warren: The main implication is that for trading in assets where price increments have a long-term correlation structure, you can get a real edge over the traditional Almgren and Chriss baseline across most market conditions.

Iris: What stands out is the robustness; even when they mess up their estimates for the Hurst exponent H and volatility sigma by twenty percent, the final risk measure only degrades by about five percent.

Ruth: That resilience is important because in real markets, you never have perfect knowledge of how much memory an asset has or its exact volatility.

Warren: So to summarize, they developed a policy gradient algorithm that learns an optimal liquidation strategy under fractional Brownian motion dynamics by maximizing CARA utility, and it shows this method beats the Almgren and Chriss baseline for almost all scenarios.

Iris: The work relies on this specific structure—the variation doing the work is this learned policy function parameterized by a multilayer perceptron—which is what makes the identification solid.

Ruth: It's a neat piece of engineering because it solves a continuous control problem by breaking time into discrete intervals and using that neural network setup.

Warren: And for anyone listening who only cares about trading, this means if you’re dealing with assets that exhibit long-range dependence, you should probably stop thinking about simple Brownian motion models for your execution planning.

Iris: It does open up a lot of questions about what happens when we move beyond linear market impact assumptions and try to incorporate more complex friction costs into the AI's learning process.

Ruth: That’s exactly where we need to go next, exploring how this framework could adapt if the cost of trading shares wasn't just a simple linear function of the quantity sold.

Matthieu Garcina, Jiang Pua Thomas Verriera

q-fin.TR, q-fin.CP, q-fin.ST

Submitted: 2026-11-08

Updated: 2026-11-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: Optimal liquidation in a fractional setting with reinforcement learning develops a strategy to exploit information induced by the correlation between price increments…

Key concepts

Fractional Brownian Motion (fBm)
This is a mathematical model describing asset price movements where the increments exhibit long-range dependence, meaning past price changes significantly influence future changes. Unlike standard models, fBm captures the memory and persistence of market fluctuations over time.
CARA Utility
Constant Absolute Risk Aversion (CARA) utility is a mathematical framework used to define what constitutes an 'optimal' trading strategy. It measures risk aversion based on the absolute dollar change in wealth rather than just variance, guiding the AI to make decisions that balance potential gains against the certainty of losses.
Policy Gradient Algorithm
This is a machine learning technique where a neural network learns an optimal action (like how much to sell) by interacting with an environment. It adjusts the network's internal parameters iteratively to maximize a defined reward, allowing it to discover complex, dynamic control policies.
Hurst Exponent (H)
The Hurst exponent is a key parameter that quantifies the long-term memory or persistence in the asset's price changes. If H > 0.5, movements are persistent (trend-following); if H < 0.5, they are anti-persistent (mean-reverting). The model specifically adapts its strategy based on this value.

Terminology

Summary

Optimal liquidation in a fractional setting with reinforcement learning develops a strategy to exploit information induced by the correlation between price increments, outperforming deterministic solutions when asset dynamics follow a fractional Brownian motion This strategy is derived from a policy gradient algorithm parameterized by a neural network trained to maximize the CARA utility

Problem Formulation and Motivation

The optimal liquidation problem considers selling a quantity Q of shares over time T with linear market impact, but for fractional Brownian motion, the deterministic solution is no longer optimal The paper focuses on maximizing the constant absolute risk aversion (CARA) utility defined by formula (2), which allows for the application of Bellman’s principle The authors introduce a policy gradient algorithm parameterized by a neural network to numerically approximate the optimal dynamic controls under this criterion

Model and Feature Engineering

The model uses an implementation shortfall benchmark where the payoff is defined as XT − QS0, and the objective is to solve the control problem fined by formula (2) The features fed to the network are denoted as Ft = (t, qt, St, Rˆt,t+τ) where Rˆt,t+τ is the conditional expectation of the future price return over a horizon τ The optimal lags for the forecast are computed via numerical optimization given equation (7), which depends only on H

Training Algorithm and Policy Gradient

The policy gradient algorithm is designed for a time discrete problem, dividing [0, T] into N equal intervals at times (tk = kT/N) The policy is parameterized by πθ(q F) = N µθ(F), σ2explo, where µθ(F) is a multilayer perceptron The reward function used in the reinforcement learning setup is defined as Gnn = − exp (−λΠnnT), and the objective is to maximize J(θ) = Eπθ Gnn A modified gradient approximation is used to reduce variance by comparing the learned policy to the AC baseline strategy

Performance and Insights

The learned policy outperforms the Almgren-Chriss baseline for every H ̸= 1/2 and recovers it at H = 1/2 The gain is larger when H is far from 1/2, with an asymmetry favoring the anti-persistent regime The visualization of individual trajectories reveals two qualitatively distinct behaviors: trend-following with short positions when increments are persistent, and inventory-holding to capture mean-reversion when they are anti-persistent The method is robust to estimation error in (H, σ), as even a 20% relative error on (H, σ) only costs about 5% of the certainty equivalent

Conclusion

The developed policy gradient algorithm learns a liquidation strategy under fBm dynamics that outperforms the AC baseline for every tested value of H ̸= 1/2 The gain grows as H moves away from 1/2 and is markedly larger in the anti-persistent regime A moderate number of lags suffices, with most of the gain captured at p = 5 The method is robust to parameter misspecification

References

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Policy gradient algorithm for optimal liquidation under fractional Brownian motion: The system can learn an execution strategy conditioned on a vector of features Ft = (t, qt, St, Rhat t t + tau) to maximize CARA utility.

  2. Exploiting serial correlation via feature engineering: The system can use lagged returns Zt = (Rt - delta1 t - delta0 t,..., Rt - delta p t - delta p-1 t) and a covariance-based predictor Rhat t t + tau to condition its actions on past price increments.

  3. H specific model training: The system must be trained with a distinct model for each value of the Hurst exponent H because the entire pipeline is H-specific: the Hurst exponent must be known both to simulate the training trajectories and to compute forecasts.

  4. Robustness to parameter misspecification: The system can maintain high performance even if the Hurst exponent H and volatility sigma are estimated incorrectly, as demonstrated by Table 3 showing that a 20% relative error on (H, sigma) only degrades the certainty equivalent by about 5%.

  5. Qualitative behavior learning: For persistent regimes (H > 1/2), the system learns to exhibit trend-following with short positions when increments are correlated, and in anti-persistent regimes (H < 1/2), it learns inventory-holding and mean-reversion capture.

Related papers