Reinforcement Learning to Harness Approximation Errors for Long-Time Quantum Simulation

The gist: RL-Trotter is a reinforcement learning framework that treats unavoidable approximation errors as resources for error correction rather than merely imperfections to suppress, discovering self-correcting sequences in which later errors compensate for those accumulated earlier, increasing the accuracy of the long-time dynamics

How it works

The RL-Trotter framework is designed to adaptively correct accumulated errors in energy and energy variance throughout the entire evolution by selecting a single scalar—the next Trotter step size—as its action The agent learns to combine these errors into a self-correcting sequence whose accumulated errors remain controlled over long times This process is formulated as a sequential Markov decision process where the Trotter step size at each step is chosen according to the current evolved state The observation On fed to the agent includes conserved quantities such as the energy and its variance, together with their initially measured values and tn The action A n is the time step size dt n, chosen within a predefined range (dtmin, dtmax)

The reward function R n is designed to favor large time steps while penalizing errors in the conserved quantities The explicit form of the reward is R n = h(1 − α) exp − β1 (E n+1−E0) 2 + α exp − β2 (δE 2 n+1−δE 2 0) 2 i tanh(η dt n) The factor tanh(η dt n) rewards larger steps, while the exponential terms suppress actions that generate excessive deviations in the energy density or energy-variance density This multiplicative structure couples the error-dependent term to the step-size-dependent term, such that a large reward can be obtained only when the agent simultaneously maintains small Trotter errors and selects a sufficiently large step size

Key Properties and Advantages

RL-Trotter exhibits two crucial properties absent in state-of-the-art adaptive algorithms: intrinsic robustness against measurement noise and a strong generalization capability The intrinsic robustness directly improves training efficiency as it relaxes the precision required for estimating the observables Furthermore, the agent only observes a small, fixed set of observables—the energy and energy variance—so its input dimension does not grow with the Hilbert-space dimension, which substantially reduces measurement shots required for determining a corrective policy

The generalization capability manifests in two distinct ways:

1. A policy trained on a small, classically simulable system can be transferred directly to systems an order of magnitude larger

2. A learned policy applies to previously unseen initial states that share similar physical properties with those used in training without further optimization This enables a practical protocol where RL-Trotter is pretrained classically and then deployed on a large quantum processor, either directly or with limited fine-tuning

Benchmarking and Performance

In the benchmark of the one-dimensional mixedfield Ising chain, RL-Trotter produces significantly smaller errors in energy and energy variance density compared to conventional Trotterization, while achieving a longer total evolution time Specifically, for the quench process studied, RL-Trotter generates accurate dynamics of local observables like magnetization in the x direction at an order of 10−2 to 10−3 The average errors for energy and energy variance density are approximately one order of magnitude smaller than those of conventional Trotter

Robustness and Generalization

The robustness against measurement noise is demonstrated by training the agent in a noisy environment with relatively strong noise, such as sigma1=sigma2=0.1, and evaluating the learned policy using a weaker but finite noise strength, sigma1=sigma2=0.03 RL-Trotter remains stable against noise throughout the evolution, whereas ADA-Trotter exhibits larger fluctuations in step size and freezes under noise The generalization capability is shown by applying a policy trained on L=8 spins directly to systems with L=12, 16, and 20 without further training This transferability holds even when the post-quench Hamiltonian is fixed at different parameters, provided the initial states share similar physical properties such as the staggered order parameter

Conclusion and Outlook

The work establishes a new perspective for DQS and quantum algorithms, showing that unavoidable errors in approximate quantum evolution can be orchestrated to self-correct RL-Trotter is particularly suitable as it does not require access to the complex many-body wavefunction but only the energy and energy variance, which can be efficiently estimated using randomized measurement techniques such as classical shadows The generalization capability suggests a practical protocol where RL-Trotter is pretrained classically and then deployed on a large quantum processor Future work could involve incorporating globally conserved charges associated with symmetries or expanding the action space beyond the Trotter step size to adaptively select different DQS algorithms The framework is also applicable to other quantum algorithms where problem-specific physical or algorithmic constraints can define the observations and rewards guiding learning

How it works (DDPG)

The Deep Deterministic Policy Gradient (DDPG) algorithm is used to implement the RL framework for adaptive Trotterization, employing an actor network to map observation O n to a deterministic action A n, which is the Trotter step size dt n The critic network estimates the expected cumulative reward associated with an observation–action pair Q(O n, A n) The training involves sampling transitions D = (O n, A n, O n+1, R n, d n) from a replay buffer and updating the actor and critic networks using the Bellman equation to minimize the mean-squared Bellman error The agent aims to maximize the expected discounted cumulative reward over many Trotter steps by maximizing J = E[G0]

Computational Resources and Motivation

The training simulations were performed on a single NVIDIA A100 PCIe GPU or a single NVIDIA H200 NVL GPU, completing the process within approximately 2 hours The reward design is motivated by the need to favor large time steps while penalizing errors in conserved quantities, as the multiplicative structure couples these objectives to ensure a balance between simulation time and accuracy This reward function is compatible with realistic quantum experiments because the observables entering it, energy and its variance, are experimentally accessible without requiring full-state tomography The performance of RL-Trotter shows that the learned policy achieves a more favorable trade-off between accessible evolution time and simulation accuracy than the local greedy policy employed by ADA-Trotter

Supplementary Results

Supplementary results confirm that RL-Trotter remains robust against measurement noise, achieving longer evolution times than ADA-Trotter under noisy conditions Furthermore, the comparison of ADA-Trotter with different error tolerances shows that while relaxing constraints extends the accessible evolution time, it produces larger deviations in energy density compared to RL-Trotter The results also confirm that the learned policy can accurately reproduce the dynamics of the staggered magnetization for both system sizes, demonstrating its generalization ability The comparison between conventional Trotter and RL-Trotter rewards shows that the learned policy can choose an adaptive sequence of Trotter step sizes that achieves a substantially larger total reward than any fixed-step conventional Trotter protocol<ref:2608.

1. The RL-Trotter framework enables accurate long-time quantum simulation by treating approximation errors as resources for error correction, allowing it to discover self-correcting sequences in which later errors compensate for those accumulated earlier. This allows the improved AI system to achieve longer total evolution time while keeping violations of the conserved quantities small, enabling practical protocols that are resource-efficient by reducing the need for costly state tomography.

2. The system can be trained on a small, classically simulable system and then transfer directly to systems an order of magnitude larger, significantly reducing computational overhead; this allows for the deployment of complex simulation policies onto large quantum hardware without retraining.

3. The agent's policy is intrinsically robust to measurement noise because it optimizes the expected cumulative reward rather than the reward obtained in a single episode, which enables it to avoid overreacting to stochastic fluctuations in the measured conservation-law violations.

4. The system can be generalized across different initial states and Hamiltonian parameters by learning a policy that is physically transferable if they share the same underlying physical properties, meaning the protocol does not need redesign for each new simulation setup.

5. The reward function, defined as R n = h(1−α)exp(-β1(E n+1−E0) squared + αexp(-β2(ΔE 2 n+1-ΔE 2 0))i tanh(ηδt n), explicitly balances maximizing time and minimizing conservation errors, allowing the policy to orchestrate errors into resources for accurate and resource-efficient quantum dynamics.

6. The DDPG implementation uses an off-policy actor-critic structure with a replay buffer, which improves data efficiency for optimization problems by reusing past experience, making it suitable for the costly experimental feedback loops in quantum control.