Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting

summary

The gist

Fine-tuned generative models pretrained on Level-3 order-flow data can be fine-tuned to predict future microprice and volatility…

In short

A generative model pretrained on Level-3 order-flow data from crypto exchanges was fine-tuned to predict future microprice and volatility. The model outperformed classical methods in sub-second forecasting and showed zero-shot transfer capabilities, meaning it could predict assets it wasn't trained on, suggesting a powerful new approach for high-frequency trading signals.

Key concepts

Level-3 Order Flow Data
This is the raw data used to train the model. It records every single order in the limit order book, including details like action (buy/sell), side, price difference ($\delta p$), log volume ($\log v$), and time difference ($\Delta t$). This detailed data captures the microstructure of how trades are executed on exchanges.
Tokenization
The process of converting complex order book events into a sequence of tokens that a generative model can understand. Each individual order is converted into one token, allowing the model to learn patterns across sequences of trades rather than just single points in time.
LoRA Fine-Tuning
A technique used to efficiently adapt a large, pre-trained generative model to a specific task. Instead of retraining the entire massive model, LoRA introduces small, trainable matrices that are added to the existing model layers. This allows for rapid and effective fine-tuning on specific prediction tasks like forecasting microprice moments.
Zero-Shot Asset Transfer
The ability of the fine-tuned model to successfully predict outcomes for assets it was never explicitly trained on during its final tuning phase. This demonstrates that the learned underlying representation of order flow is general enough to apply effectively across different cryptocurrencies like BTC, SOL, and ETH.

Terminology used across episodes

This episode discusses

The paper

Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting · Read on arXiv

Fedor Shabashev

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: We're looking at the paper: "Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting".

Warren: Here's the gist of the paper. Fine-tuned generative models pretrained on Level-3 order-flow data can be fine-tuned to predict future microprice and volatility, outperforming classical baselines in sub-second forecasting and demonstrating zero-shot asset transfer capabilities Model Pretraining and Tokenization The approach involves pretraining a 300M-parameter generative model on Level-3 order-flow data collected from two crypto exchanges, Bitstamp and Bitfinex The model is trained to predict the next token with a cross-entropy loss on 2.8 billion training tokens Tokenization converts one order into one token, such as "ten buy limit orders will be converted into ten tokens in the sequence" The tokenizer uses fields like action, side, δp, log v, and ∆t at radices (2, 2, 16, 16, 16) with a vocabulary of 8192 elements Crypto required three departures from the equities recipe due to bursty activity and trade modifications Fine-Tuning Methodology The pretrained model is fine-tuned via LoRA to predict the first and second moments of the future microprice on BTC/USD across horizons from 200 ms to 30 seconds The fine-tuning uses a decoupled loss Huber(µ, r) + Huber(log σ, log rv) instead of a naive NLL because the latter causes the mean to collapse The model is compared against classical benchmarks such as gradient boosting and GARCH for predicting microprice change direction and volatility Performance on Sub-Second Horizons The fine-tuned model outperforms classical baselines in predicting the microprice change direction at the 200 ms and 500 ms horizons Furthermore, it outperforms GARCH on volatility in four of five evaluation windows The model beats order-book imbalance (OBI), including a 10-level book and OBI combined with recent-return momentum, at every sub-second horizon The margin against the gradient-boosted baseline is steady, reaching +0.05–+0.08 in correlation across sub-second horizons Zero-Shot Transfer and Generalization The predictive ability of the model transfers zero-shot to assets it was never fine-tuned on, specifically beating basic baselines for SOL and ETH The backbone is pretrained on 73 asset streams across both venues, so the representation has seen their order flow, but the LoRA and head are fine-tuned on BTC alone The fine-tuned mapping is thus asset-general, with liquidity setting how short the horizon can be pushed Inference Latency and Practical Use The end-to-end inference latency for a streaming system is 4.0 ms on the H100, which is 2% of the shortest horizon forecasted and sustains 247 events/s against the median rate of this instrument The signal is a real, leakage-free, asset-general prediction, but not directional alpha The plausible uses are in execution (anticipating imminent adverse selection) rather than taking The fine-tuned model beats gradient boosting on order-flow features, and the edge is a short-horizon forecast that forecasts the imminent order-flow continuation rather than a persistent multi-horizon signal The barrier to tradeability is one of move magnitude, not forecast accuracy Limitations include fine-tuning on a single asset and a single two-day window, which limits regime coverage The volatility comparison shows the second-moment claim is regime-conditional The model’s edge forecasts the imminent order-flow continuation rather than a persistent multi-horizon signal Future Work Future work suggests pretraining on more data, per-asset fine-tunes, and larger models might improve accuracy The state of the order book is potentially useful information, hence encoding it and feeding it to the model might improve the accuracy A longer context window might lead to higher accuracy for pairs with a lot of transactions Modifications to the architecture like linear attention or sliding-window attention might be helpful once a longer context window is achieved References Fernando Acero What do foundation models for market microstructure learn?

Ruth: A controlled study with synthetic limit order books.

Warren: arXiv preprint arXiv:2610.20104, 2026 Jakob Albers Mihai Cucuringu Sam Howison, and Alexander Y Shestopaloff The good, the bad, and latency: exploratory trading on Bybit and Binance.

Ruth: Quantitative Finance, 25(6):919–947, 2025 Abdul Fatir Ansari et al.

Paper summary: Ruth: So looking at the whole thing, the paper "Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting" is essentially about taking a large generative model and tailoring it to predict very short-term price and volatility changes using order flow data.

Warren: The authors claim their fine-tuned model beats classical methods on prediction direction at two hundred millisecond and five hundred millisecond horizons, and it also shows zero-shot transfer to new assets, which is a pretty big part of the contribution <ref:2611.14117#pg2>.

Iris: It seems like the main point here is that they've managed to fine-tune this foundation model for predicting both the first and second moments of microprice using a specific loss function that handles uncertainty well.

Ruth: The implication is that instead of just looking at handcrafted features, we can use these large models to capture order flow dynamics in a way that outperforms things like gradient boosting on those features.

Warren: And while it's useful for sub-second execution, the authors are honest about the limitations, noting that they only fine-tuned on one asset and two days of data, which means the regime coverage is pretty limited.

Iris: And they also clarify that the claim about beating volatility is actually conditional on specific market regimes because it's not universally true across all conditions.

Ruth: So in simple terms, this paper shows how a generative foundation model can be fine-tuned to predict very short-term price moves based on order flow, offering a tool for execution and making things more efficient in the market.

Conclusion: Ruth: So we’ve seen how they took this huge generative model and fine-tuned it specifically for order flow predictions on BTC, looking at how they framed that in "Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting."

Warren: Yeah, I think the authors are saying that this isn't just another fancy trading tool; it’s about getting a model to actually learn the structure of the order book itself.

Iris: What they're doing is taking data from those two exchanges and using it to train a model that predicts both the next price and how much volatility there will be over very short periods, like two hundred milliseconds or five hundred milliseconds.

Ruth: Exactly, and what’s interesting is that they use this specific way of training—decoupling the loss function—to make sure the model doesn't just guess a number, but actually captures the uncertainty in those price changes correctly.

Warren: And on the results side, they show it beats classical stuff like gradient boosting and GARCH for predicting whether the price is going up or down at those sub-second levels.

Iris: They also managed to get this model to work on assets it hadn't seen before—like SOL and ETH—even though they only fine-tuned the part dealing with Bitcoin. That’s a pretty strong generalization claim there.

Ruth: It suggests that the way this model learns order flow patterns is actually kind of general, not just specific to one coin or one set of trading conditions.

Warren: But we gotta remember the caveats they laid out in their conclusion—the edge they find is really tied to that very short timeframe, which means it’s forecasting what's about to happen right now, not a long-term trend.

Iris: And they also flagged that the claim about beating volatility isn't solid across all market situations; it depends on the specific regime you are in.

Ruth: So the real implication here is that this AI approach can provide a very fast, real-time signal for execution, but we have to be careful about overinterpreting it as a persistent alpha generator.

Warren: It’s definitely a tool for anticipating immediate adverse selection during trades, but the barrier to actually using it is how much movement you are expecting.

Iris: If we look at what they suggest next—pretraining on more data and maybe incorporating the actual state of the order book—that could definitely push those accuracy numbers further.

More episodes

← Home