Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion

arXiv:2610.03830 · cs.LG · Submitted 2026-10-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion".

Tom: The memory-state critic introduces a novel asymmetric actor-critic method that conditions on both the state and the policy's internal memory,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: We just talked about how this paper, "Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion," is focused on fixing a problem where critics that only look at the state fail in partially observable Markov decision processes >

Jane: Right. The core of the issue is that when you only condition on the state, it's generally ill-defined because two different histories leading to the same state can result in different future action distributions >

Lu: So, they introduce this new idea: conditioning the critic on both the state and the policy's own memory, which they define as the internal representation of history through which the policy selects its actions >

Meng: Think about it like this: instead of trying to perfectly encode every single thing that happened in order to define value, you just give it a summary of what it remembers from its past decisions >

Lalam: And they prove that this formulation is well-defined because the memory, when paired with the state, acts as a sufficient statistic for the future execution of the POMDP >

Tom: So, what's the big payoff? They show that this memory-state critic gives unbiased policy gradients and maintains the same on-policy value function as existing history-state critics >

Jane: That’s important because it means we don't need to worry about biasing our learning process when using this approach, which is a major step forward for applying these methods in vision-based tasks >

Lu: They also show that this method outperforms the history-state critic in vision-based pursuit-evasion environments and converges faster >

Meng: That practical performance improvement is what engineers care about; faster convergence means we get better results with less training time on hardware or simulation resources >

Lalam: It’s a nice result because it shows that this approach of conditioning on state and memory is already well-defined, which was the main hurdle in applying this method to complex POMDPs >

Conclusion: Tom: So, wrapping up this discussion on "Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion," the authors are showing us how to structure the critic for recurrent actors in POMDPs more efficiently >

Jane: Essentially, they’ve shown that we can use a single recurrent encoder instead of needing two separate ones—one for history and one for state conditioning—to get unbiased policy gradients >

Lu: The implication is that you can achieve the same theoretical guarantees as using a full history-state critic but with less computational overhead during training >

Meng: For someone building an AI system, this translates to a simpler architecture that still captures the necessary information about the environment's hidden state effectively >

Lalam: It suggests that conditioning on state and memory is a powerful way to manage uncertainty in sequential decision-making without having to track every single past observation explicitly >

Tom: So, while they didn't solve every problem out there, they’ve given us a cleaner way to handle the history dependency in these complex AI tasks >

University of Liège · KTH Royal Institute of Technology, Sweden · McGill University & Mila Québec AI Institute

cs.LG

Submitted: 2026-10-02

Updated: 2026-10-12

Comments: Accepted at the 19th European Workshop on Reinforcement Learning (EWRL 2026), Lille, France. Non-archival workshop

Code: https://github.com/LouetteArthur/MemoryStateCritic

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The memory-state critic introduces a novel asymmetric actor-critic method that conditions on both the state and the policy's internal memory, proving it yields unbiased policy gradients and

Key concepts

POMDP: Partially observable Markov decision processes model real-world problems where an agent must make decisions based on incomplete observations. The agent doesn't know the true state directly, requiring it to maintain a history of past observations and actions to make informed choices.
History-State Critic: This critic conditions on both the current state and the entire history of past states. It is well-defined because the combination of history and state forms a Markovian variable for future execution. However, implementing this requires an additional recurrent network to encode the history.
Memory-State Critic: This novel critic conditions only on the current state and the policy's internal memory (hidden state). The paper proves that this is sufficient because 'memory is a sufficient statistic of the history for future actions,' allowing it to provide unbiased policy gradients with only one recurrent encoder.
Asymmetric Actor-Critic: This refers to an actor-critic setup where the critic and actor might have different architectures or update mechanisms. The memory-state critic specifically uses this structure to condition the critic on state and memory, aiming for a more efficient learning process than previous history-based critics.

Terminology

Actor — 1× in this paper · explained in 3 episodes

Summary

The memory-state critic introduces a novel asymmetric actor-critic method that conditions on both the state and the policy's internal memory, proving it yields unbiased policy gradients and outperforms existing history-state critics in vision-based pursuit-evasion tasks.

The gist: The memory-state critic gives the same on-policy value function and the same unbiased policy gradient as the history-state critic, removing the need for a second recurrent approximator of the history.

Background on POMDPs and Policy Gradients

In many real-world decision problems, such as robotic control problems, an agent must take actions based on noisy or incomplete observations of the underlying state of its environment. Such problems are modelled as partially observable Markov decision processes (POMDPs) [Åström, 1965]. In POMDPs, the history-dependent strategy, or policy, is usually implemented with a recurrent neural network (RNN) that recurrently updates an internal state, called a hidden state, based on successive observations and actions [Ni et al., 2022]. The hidden state is trained by backpropagating the RL loss through time, not by predicting states or observations.

The History-State Critic and its Limitations

Baisero and Amato [2022] proved that a critic that conditions only on the state is generally ill-defined in a POMDP. Instead, the history-state critic, which takes as input both the history and the state, is well-defined because the history and state together are a Markovian variable of the future execution of the POMDP. However, in practice, it has to encode it with an encoder such as a recurrent network of its own. This doubles the recurrent computation at training time and induces a second network.

The Memory-State Critic Mechanism

In this work, we introduce the memory-state critic, which conditions on the state and the policy’s own memory. The critic takes as input the policy’s memory, but the critic loss is not backpropagated through the recurrent encoder of the history. We prove that this approach gives unbiased policy gradients by showing that the memory is a sufficient statistic of the history for future actions.

Experimental Results

The memory-state critic was evaluated in a vision-based pursuit-evasion environment between two quadrotors. The results show that the memory-state critic outperforms the history-state critic and converges faster. In addition to being unbiased compared to the state-only critic, it maintains a slight edge in the wall arena, where the actor’s history carries information that the privileged state alone does not.

Conclusion

We introduced a single-encoder asymmetric critic for recurrent actors in POMDPs. It uses the actor’s recurrent hidden state, which avoids learning a second statistic of the history for the value function. We proved that it gives unbiased policy gradient as well as the same on-policy value function as the history-state critic of Baisero and Amato [2022], with one recurrent encoder instead of two. This outcome demonstrates that conditioning on the state and the actor’s memory keeps the policy gradient unbiased<ref:2610.03830/pg2, However, in the open arena, the evader is visible whenever it is in frame, and the observation model is deterministic, so the state is close to Markov for the future return, and the state-only critic is nearly unbiased<ref:2610.03830#pg8>. The memory-state critic achieves the best capture rate in the wall arena.

Appendix Details

The quadrotor dynamics model each quadrotor as a rigid body of mass m and inertia tensor J, with state (p, v, R, ω, omega) for position, velocity, attitude, body rates, and the four rotor speeds. The policy outputs a four-dimensional action a ∈ [−1, 1]4 at 25 Hz<ref:2610.03830/pg11, The policy outputs a four-dimensional action a ∈ [−1, 1]4 at 25 Hz>. The reward coefficients in (9) are R = 10.0, κa = 3.0, d0 = 0.5 m, κp = 0.02, κρ = 1.0, and κbr = 0.01<ref:2610.

Improvements for AI systems

This improves policy gradient stability by replacing a potentially ill-defined history-state critic with a memory-state critic, which gives unbiased policy gradients by conditioning on the state and the policy’s own memory.
The improved system can learn complex, history-dependent control policies in partially observable Markov decision processes (POMDPs) without needing a second recurrent approximator for the history, as it leverages the actor's internal representation directly.
By using the memory-state critic, the agent is able to maintain an unbiased estimate of the value function, ensuring that Vπ(st, za) is an unbiased estimate of the history value function and that all advantages are equivalent to those derived from a history-state critic.
In environments with high partial observability, such as the wall arena where the state aliases the history, this system maintains a marginal performance edge over baselines, suggesting better handling of ambiguity in occluded states.

Abstract

In partially observable Markov decision processes, the optimal policy generally depends on the history of observations and past actions. Asymmetric actor-critic methods have become popular to learn such policies when additional information, such as the true state of the environment, is available during training. The critic, which is not needed at execution, is given access to the state. A critic conditioned on the state alone is generally ill-defined and yields biased policy gradients. Conditioning on the state and the history, the history-state critic restores both. In this paper, we show that conditioning the critic on the state and the policy's own memory, i.e., the internal representation of the history through which the policy selects its actions, is already well-defined and gives unbiased policy gradients, removing the need for a second recurrent approximator of the history. We call it the memory state critic. It follows that a critic based on the policy's memory need not backpropagate its loss into that memory, even though the memory is a lossy encoding of the history. We evaluate the memory-state critic in a vision-based pursuit-evasion environment between two quadrotors across two arena types. The pursuer is the learning agent, and the evader is sampled per episode from a fixed pool of heuristic behaviours. The results show that the memory-state critic outperforms the history-state critic and converges faster. In addition to being unbiased compared to the state-only critic, it maintains a slight edge in the wall arena, where the actor's history carries information that the privileged state alone does not.