Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies
The paper links recurrent neural network hidden states to Pontryagin's minimum principle in optimal control theory. It shows that these hidden states implicitly track the co-state, which represents sensitivity to state changes. This connection enables a new training method where the critic's gradient directly shapes the actor's hidden state toward optimal control performance.