Reinforcement Twinning via Twin-Calibrated Bayesian Policy Search and Replay-Driven Identification
eess.SY, cs.SY
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: The feasibility of updating and synchronising digital twins of actuated systems from operational data depends strongly on how the system is controlled and, consequently…
Terminology
Abstract
The feasibility of updating and synchronising digital twins of actuated systems from operational data depends strongly on how the system is controlled and, consequently, on the data generated during operation. This establishes a direct link between control actuation and model identifiability. This work presents a reinforcement-twinning framework that exploits this link through three computational components. A Gaussian-process surrogate of the physical return combines the twin prediction, used as a calibrated trend, with a discrepancy learned from measured returns, allowing Bayesian policy search to exploit the twin without being tied to its predictions. Candidate policies are proposed around the policy that is optimal for the twin, while measured returns can expand this region. The resulting physical episodes are replayed on the twin after observer-based synchronisation with sparse measurements, and its parameters are identified by combining Bayesian optimisation with gradient-based refinement. Selected information-oriented episodes deploy an excitation policy to improve parameter identifiability. The framework is demonstrated on a Burgers-flow analogue of dynamic-induction control in wind farms, with a deliberately misspecified model structure. Despite substantially biased return predictions, the twin provides a useful starting policy and search region, placing the guided search ahead of unguided Bayesian optimisation from its first physical evaluations. This advantage is maintained when the search region is allowed to expand in response to changes in measured returns, whereas searches over-constrained by the twin-defined region are eventually overtaken by the unguided alternatives. The results support reinforcement twinning as a framework for jointly adapting models and control policies when physical interaction is limited and model discrepancy cannot be neglected.
Sources
- Sequential Bayesian optimal experimental design via approximate dynamic programming
- MBMF: Model-Based Priors for Model-Free Reinforcement Learning
- Reinforcement Twinning for Hybrid Control of Flapping-Wing Drones
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction