Mixture-of-Experts for Cryptocurrency Order Execution: Training Stability, Tail Risk, and Failure Modes

The gist The mixture-of-experts approach, combining K-Means partitioning with Double Deep QLearning experts, does not improve mean execution cost over vanilla DDQL, but reveals that training specifications significantly impact training stability and tail risk in cryptocurrency order execution environments.

Execution Cost and Baseline Performance

Execution costs separate an investment strategy’s hypothetical returns from its realised ones: commissions, limited book liquidity and reactive pricing open a gap that Perold [25] formalised as the implementation shortfall. The classical response schedules a parent order into child orders over time, equally sized (TWAP) or volumeproportional (VWAP), refined by the optimal execution literature [3, 5] and, given the problem’s dynamic-programming structure and high-dimensional book state, by reinforcement learning (RL) from Nevmyvaka et al. [22] through modern deep Q-learning variants [19, 23]. Most of this literature was developed for public equity markets Cryptocurrency markets differ along dimensions that matter for execution: continuous trading without auctions, fragmented and comparatively thin liquidity, retail-dominated flow, and rapidly shifting volatility and spread regimes [1, 10, 34].

Mixture of Experts Architecture

The Mixture of Experts (MoE) paradigm [15, 30] maintains several policies behind a router that dispatches each situation to one of them. The MoE architecture consists of two primary components: (1) Expert Q-Networks (E1,..., EK): A set of K independent DDQL agents, each with the same 5-layer fully connected architecture as the vanilla DDQL model. (2) K-Means Router: A K-Means clustering model fitted on episode-level market features (K=4 in the default configuration; we vary K ∈ 6, 8). At inference time, to strictly prevent look-ahead bias, the features of the previous episode are mapped to the nearest cluster centre. Under hard gating the expert for episode e is the centre nearest the previous episode’s standardised feature vector, which is what enforces causality.

Training Stability and Failure Modes

The paper evaluates vanilla Double Deep QLearning (DDQL), K-means-partitioned mixtures of DDQL experts at K ∈ 6, 8, and dense networks parameter-matched to the K=4 and K=8 expert budgets. No learned configuration significantly improves mean implementation shortfall over DDQL. Acrossseed dispersion is lowest at K=8 but non-monotone and not robust to family-wise adjustment, while within-policy tail risk worsens monotonically with K. The apparent attribution of the failure mode reverses between 30 and 100 seeds, illustrating the importance of repeated-seed evaluation.

Specification Decomposition

The authors investigate how specification changes interact non-additively. They test three specification choices implicated in the failure mode: a reward that differs from the evaluation metric, a non-annealed exploration schedule ending at epsilon ≈ 0.88, and asymmetric replay-buffer retention across arms. Annealing suppresses collapse under the as-reported reward. Combining annealed exploration with the aligned reward restores collapse in 19/30 runs; with all three specification changes, it rises to 48/100, revealing a strong non-additive interaction.

Key Findings on Stability and Risk

The authors report that across-seed dispersion is lowest at K=8 but non-monotone and not robust to family-wise adjustment, while within-policy tail risk worsens monotonically with K. No MoE K=8 run collapses under any of the six specifications tested. The reference rows of Table 2 record a fact we will not omit: on identical episodes and fill mechanics, TWAP beats every learned configuration on mean IS (0.39 bps) and tail risk (CVaR-95 74.0), better than every agent, and immediate liquidation 0.21 bps. Annealing alone removes the collapse and brings DDQL to 0.28 bps, MoE K=8 to 0.01 and DDQL-cap8 to 0.19, all below TWAP, with per-run CVaR-95 at 31–50 against TWAP’s 74.0. This is not evidence that RL execution works: 0.01 bps is execution at essentially the arrival price, attainable only because the replay is frictionless. The correct reading remains diagnostic: the architectures reshape the distribution of training outcomes, while the training specification determines whether any of them clears a schedule baseline.

Limitations

First, the historical replay is frictionless: the agent’s own trades have no feedback on subsequent prices or book states, each bar’s book is a 5-minute mean-aggregated book rather than an instantaneous one. Second, the environment omits the nonlinear and transient impact documented in the microstructure literature [2, 13, 24]. Third, the state omits richer features such as multi-level depth imbalance. Fourth, the sample is a single instrument over two weeks, with evaluation on one. Fifth, the model does not represent macroeconomic shocks or flash crashes [12]. Sixth, every comparison rests on a single fixed draw of 500 evaluation windows shared across models, so pooled tail statistics contain 500 distinct market scenarios, not 50,000 independent ones. Seventh, the training reward is not the metric reported at evaluation, a mismatch with real consequences; combined with frictionless replay and the terminal rule, it makes the environment’s own optimal policy trivial in a way live markets’ is not. Eighth, those 500 windows are heterogeneous: TWAP scores +0.39 bps over the full set but −1.66 bps over the first 120 episodes, so no subset is representative and we report none. Ninth, the decomposition of Section 5.4 covers three of six configurations, half its arms use 30 rather than 100 seeds, and two of the three two-change arms were not run.

The paper is concerned with analyzing training stability and tail risk in cryptocurrency order execution using a Mixture of Experts framework to determine if architectural gains are reproducible or merely artifacts of training specifications. The gist The mixture-of-experts approach, combining K-Means partitioning with Double Deep QLearning experts, does not improve mean execution cost over vanilla DDQL, but reveals that training specifications significantly impact training stability and tail risk in cryptocurrency order execution environments. This research matters because it establishes that architectural performance gains in deep reinforcement learning policies are often reflections of favorable training realizations rather than intrinsic properties of the architecture, providing a crucial diagnostic tool for understanding policy robustness.

How it works

The MoE model trains in two stages: Warm-up stage: The agent executes episodes using a TWAPcentred epsilon-greedy policy (identical to DDQL exploration). After each episode, a four-dimensional feature vector is extracted: realised volatility (standard deviation of log-returns along the episode’s price path), an execution-cost proxy (average ratio of per-trade cost to mid-price), schedule imbalance (how front- or back-loaded the agent’s own execution was), and the impact ratio kappat /alphat, constant in this experiment and mapped to zero by standardisation, so routing is effectively three-dimensional. After collecting features from 300 warm-up episodes, KMeans (K=4, k-means++ initialisation [4]) is fitted on the z-score–normalised feature matrix. Specialisation stage: Each episode is assigned to the centre nearest the previous episode’s feature vector, the same rule used at inference. Transitions are stored only in the corresponding expert’s replay buffer, and standard DDQL updates (Double Q-learning with target network, MSE loss) are applied per expert.

Experimental Design and Metrics

The core metric is Implementation Shortfall (IS) [25]. Positive IS is a cost; lower is better. Agents train on the January 9–14 period and are evaluated exclusively on the temporally disjoint January 16–20 period. The simulation re-plays historical book data directly rather than synthesising a background population, a choice we return to in Section 6.

1. textbf Expert Partitioning for Stability Assessment: Replacing Mean Performance Claims with Diagnostic Insights. The paper suggests that expert partitioning is not necessary to suppress this failure under the as-reported specification and that the apparent attribution of the failure mode reverses between 30 and 100 seeds, illustrating the importance of repeated-seed evaluation. This allows AI systems to diagnose whether observed performance gains are due to genuine architectural superiority or merely a favourable training realisation.

2. textbf Non-Additive Specification Analysis: Identifying Robust Learning Regimes. The study found that the specification changes interact non-additively, where annealed exploration alone eliminates the observed collapse, but adding the aligned reward restores it in 19 of 30 runs. This enables AI researchers to develop more robust training protocols that identify which combination of hyperparameters (exploration schedule, reward alignment, replay retention) yields stable policies.

3. textbf Tail Risk Quantification via Capacity-Matched Controls: Distinguishing Architectural Effects from Data Dilution. The paper notes that within-policy tail risk worsens monotonically with K, and contrasts this with the capacity-matched control which shows a better tails are directional only. This allows for the design of AI systems where performance is not just measured by mean shortfall but by explicitly quantifying how expert partitioning impacts worst-case scenarios.

4. textbf Robustness to Parameter Tuning: Verifying Floor Independence. The research demonstrated that the deep-book fill-price floor never binds at this order size, showing that extrapolating off the bottom costs about 0.067 bps per level and leaves the fill at 0.9997 of mid in the worst bar, well above every phi in the 30–90% sweep. This confirms that systems can be designed with confidence that execution costs are largely insensitive to minor parameter settings like the floor percentage.