The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models
This research quantifies how much memory (state or context) sequence models need to forecast accurately under different memory laws. It proves that for algebraic decay, required state complexity scales logarithmically with desired error, and it links forecast accuracy directly to the model's stability margin, showing that high precision forces the system toward a critical state.