From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL
The paper addresses how to create Markovian states from time-series causal graphs for reinforcement learning. It proposes MOSE, a method that exposes a single Q-function to multiple state orders simultaneously (1 through W), and Causal-MOSE, which combines this with a minimal state construction. Results show that controlled redundancy—combining the minimal Markovian core with MOSE—outperforms both methods alone, proving that simply using the minimal state is insufficient for deep RL performance.