Continual Graph Multi-Agent Reinforcement Learning

arXiv:2610.10302 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Continual Graph Multi-Agent Reinforcement Learning".

Jane: The gist In Continual Graph Multi-Agent Reinforcement Learning (CGMARL), agents learn cooperative policies across sequences of tasks,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the authors and the title of this paper. The title is Continual Graph Multi-Agent Reinforcement Learning. You see how it immediately signals that this work focuses on multi-agent systems learning over time in a way that respects their underlying structure.

Jane: And I think the authors, Tom Marzi and his team, are tackling a really specific problem: adapting to new tasks while keeping old knowledge intact, which is a huge challenge in continual learning settings.

Lu: They're setting up the problem by defining it as a sequence of tasks T = M1...MK where each task Mk is defined by a graph Gk sampled from some distribution Z, which just formalizes the structural changes we talked about earlier.

Meng: So they are taking those real-world structural shifts, like different network topologies, and putting them into this mathematical framework so the AI knows how to handle that sequence of change.

Lalam: The authors are essentially saying that to solve this, we need a framework where the graph structure itself is central to the learning process across all tasks.

The paper's summary: Tom: So what is the actual mechanism they propose? Basically, they introduce Continual Graph Multi-Agent Reinforcement Learning, or CGMARL. It formalizes graph-based MARL by equipping the Dec-POMDP with a graph G = ⟨N, E, X, E⟩ where N and E are agents and edges.

Jane: That means the structure of the environment—the graph G—directly determines things like how state transitions happen and what rewards look like for each agent. It’s all about the graph shaping the dynamics.

Lu: The paper then breaks down three types of problems: Topology Switch, where connections change between tasks, Connectivity Switch where nodes stay fixed but edges change, and Graph Features Switch where the structure is static but static features can discriminate between graphs.

Meng: That’s a lot of different scenarios just to cover how task sequences can vary in practice. It shows they are thinking about more than just one type of structural change when we talk about continual learning in this context.

Lalam: The central problem they identify is that standard graph neural networks used for policies often forget because they learn local rules that don't apply when the graph dynamics change, which is what leads to forgetting.

The paper's improvements: Tom: Now let's talk about the proposed solution, which is called Frozen Graph Encoder or FROG. It’s a simple approach because it decouples graph processing from action prediction. They propose a policy where the graph encoder ψθ1 captures the structural information while an action module µθ2 directly maps those representations to actions <ref:2610.10302#pg1>.

Jane: By freezing the graph encoder, they are trying to stop that representation learning part from forgetting past structures when new tasks come along. They enforce a condition where psi k equals psi h for each task pair h and k, which is how they preserve the past structural information in graph-based CMARL policies.

Lu: That decoupling allows the action prediction module, µθ2, to continuously adapt to new tasks using standard continual learning methods without being constantly disrupted by structural changes in the representation learning part <ref:2610.10302#pg1>.

Meng: I'm looking at this idea of freezing the encoder after learning the first task. It sounds like a way to stabilize things so we don't have to re-learn everything from scratch every time, which is good for deployment stability at scale.

Lalam: The empirical validation shows that combining FROG with standard continual learning methods like Elastic Weight Consolidation or Memory Aware Synapses substantially improves performance on multiple CGMARL scenarios and makes the system less prone to forgetting.

Conclusion: Tom: So to wrap up, the paper on Continual Graph Multi-Agent Reinforcement Learning introduces CGMARL and proposes FROG to mitigate forgetting by decoupling graph processing from action prediction and freezing the encoder after learning the first task.

Jane: The implication for us is that when we face real-world multi-agent problems with shifting structures, we can use this framework to maintain performance on old tasks while still learning new ones effectively.

Lu: The paper suggests that by controlling where you freeze the graph encoder—specifically after the first task—you can achieve competitive performance against standard methods and keep plasticity.

Meng: For deployment, it means we have a more stable path because the structural part is locked down, so we don't run into catastrophic forgetting as easily when we introduce new operational conditions.

Lalam: The Continual Graph Multi-Agent Reinforcement Learning paper gives us a concrete way to handle the structural dependencies in multi-agent learning over time, and it shows that stabilizing those representations helps maintain performance across different task structures.

Tommaso Marzi, Ahmed Hendawy, Jan Peters, Carlo D’Eramo, Andrea Cini, Cesare Alippi

IDSIA USI-SUPSI University of the Italian Swiss Institute of Technology Technical University of Darmstadt Hessian AI Robotics Institute Germany German Research Center for AI University of Würzburg EPFL Politecnico di Milano

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/jselvaraaj/JaxInforMARL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 74/100

The gist: The gist In Continual Graph Multi-Agent Reinforcement Learning (CGMARL), agents learn cooperative policies across sequences of tasks, aiming to adapt effectively to new tasks while preserving the

Key concepts

CGMARL Framework: This framework treats a sequence of tasks as a series of attributed graphs. Each graph defines the environment's rules, such as how agents interact and what rewards they receive. This structure allows agents to learn cooperative policies that adapt to changing task structures over time.
Frozen Graph Encoder (FROG): FROG is a strategy designed to stop forgetting by separating the graph processing from action prediction. It freezes the initial graph encoder after learning the first task, preserving structural knowledge while allowing the action module to adapt flexibly to new tasks using standard continual learning techniques.
Topology Switch (TS) Problem: This problem type involves task sequences where connections (edges) can change between tasks. The nodes (agents) remain the same, but the specific way agents are connected in the environment varies across different learning stages. It tests how well a model handles structural changes.
Graph-based MARL: This is a standard approach where Graph Neural Networks (GNNs) are used to learn policies for multi-agent problems defined on graphs. The graph structure dictates the dynamics of the problem, meaning agents must use this topology information to make decisions and solve the task.

Terminology

Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist In Continual Graph Multi-Agent Reinforcement Learning (CGMARL), agents learn cooperative policies across sequences of tasks, aiming to adapt effectively to new tasks while preserving the ability to solve previously encountered ones. This framework addresses the challenge of continual learning in multi-agent systems by modeling task sequences as attributed graphs and proposing a method called Frozen Graph Encoder (FROG) to mitigate forgetting.

CGMARL Framework and Problem Formulation

The CGMARL framework models continual structure-based MARL problems as a series of attributed graphs, where each graph determines the environment dynamics and the number of agents for the corresponding task. A single-task formulation, Graph-based MARL, equips the Dec-POMDP with a graph G = ⟨N, E, X, E⟩ where N and E represent nodes (agents) and edges. This graph G represents the task structure and determines the environment dynamics, i.e., the transition function PG and/or reward function RG depend on G. The CGMARL problem is defined as a sequence of tasks T = M1,...,MK, where each task Mk is defined as in Eq. 2 and characterized by the graph Gk sampled from the distribution Z.

Types of CGMARL Problems

The paper formalizes three main classes of CGMARL problems based on how the task sequence varies:

Topology Switch (TS) problem, where connections (edge set Ek) remain fixed within a task, but may vary across different tasks.
Connectivity Switch (CS) problem, which is a sub-case of the TS setting in which the node set remains fixed across tasks.
Graph Features Switch (GFS) problem, where the topology (i.e., node and edge sets) remains fixed throughout the task sequence, and static graph features can discriminate between different graphs.

Failure Modes of Standard Policies

Graph Neural Networks (GNNs) used in standard graph-based MARL policies are prone to forgetting because they learn local (node-level) rules that may not directly apply to unseen tasks if the underlying graph-dependent dynamics change. Experiments on the Graph-based Formation (GRAFO) benchmark show that standard graph-based methods fail to generalize across different graphs and strongly suffer from forgetting, with some continual settings resulting in complete forgetting of previously learned knowledge.

Frozen Graph Encoder (FROG) Mitigation Strategy

To address the limitation of standard methods, the paper proposes Frozen Graph Encoder (FROG), a simple approach that relies on decoupling graph processing from action prediction. FROG leverages a frozen graph backbone to preserve past structural information in graph-based CMARL policies by imposing ψk = ψh for each task pair h, k. This decoupling facilitates analyzing forgetting associated with graph representation learning and allows the action prediction module to continuously adapt to new tasks using standard continual learning (CL) methods.

Empirical Validation and Results

Experiments on the GRAFO benchmark validate FROG’s effectiveness when combined with standard CL methods, showing that it consistently improves performance when paired with them and is less prone to forgetting. The analysis of regularization coefficients reveals that the best tradeoff is achieved with λµ = 106, which is much lower than the value associated with the graph encoder (λψ = 1010), allowing FROG to maintain plasticity while achieving competitive performance. Furthermore, comparing FROG against its untrained variant (FROG (RND)) shows that random graph features hinder learning in GRAFO, as the model struggles to learn the hardest tasks (low AP values). The results suggest that using frozen graph representations stabilizes learning and achieves competitive performance against standard counterparts.

Conclusion and Future Directions

The work introduces the CGMARL framework, identifies real-world applications induced by different task (graph) distributions, and proposes FROG to mitigate forgetting. Future research directions include a theoretical investigation of the failure modes of standard CL approaches in this setting, considering off-policy MARL algorithms for testing experience-based CL methods, and developing other CGMARL evaluation benchmarks with different practical applications. The paper also notes that task order can impact learning in standard CL methods, and FROG relies on freezing the graph encoder after training the first task to achieve competitive performance.

How it works

The CGMARL framework models continual structure-based MARL problems as a series of attributed graphs, each modeling a task-specific structure and determining the environment dynamics and the number of agents. Graph-based MARL is formulated as a Dec-POMDP where the graph G = ⟨N, E, X, E⟩ represents the task structure and determines the environment dynamics. The CGMARL problem is defined as a sequence of tasks T = M1,...,MK, where each task Mk is defined as in Eq. 2 and characterized by the graph Gk sampled from the distribution Z.

Frozen Graph Encoder (FROG) Mechanism

FROG decouples graph processing from action prediction by defining a policy πθ = µθ2◦ ψθ1, where ψθ1 is a generic GNN as in Eq. 3 and µθ2 is an action prediction module. The graph encoder ψθ1 captures the structural information, while the action module µθ2 directly maps node representations to actions. To eliminate forgetting in the graph representation learning module caused by structural changes, FROG imposes ψk = ψh for each task pair h, k. This is achieved by freezing the graph encoder after learning the first task, preserving its learned parameters while allowing it to generate distinct graph representations for each task-specific graph.

Evaluation Metrics

The models are evaluated using metrics from standard CL literature, including Average Final Performance (AFP), Average Forgetting (AF), and Average Plasticity (AP). AFP measures the model’s performance across all tasks after learning the last one. AF measures how performance on past tasks deteriorated after learning the final task (backward transfer). AP measures the performance on a task immediately after learning it.

GRAFO Benchmark

Graph-based Formation (GRAFO) is a novel graph-based navigation benchmark that supports generating arbitrarily long CGMARL sequences, where agents navigate to a target configuration determined by the static input graph characterizing the task. In GRAFO, agents must leverage structural information to identify their targets because other agents’ targets are not directly observable and are implicitly defined by the graph topology.

Task Sequence Definitions

CGMARL sequences can be induced by varying the task specification in Eq. 9, which defines a series of different graphs. Examples include the Topology Switch (TS) sequence, where different tasks are specified by different topologies, and the Connectivity Switch (CS) sequence, where different tasks are specified by different edges with a fixed node set. The Graph Features Switch (GFS) scenario varies only static graph features while keeping the topology fixed.

Implementation Details

The GRAFO benchmark is implemented on top of the Jax version of InforMARL [52], and continual learning methods are adapted from the MEAL benchmark. Hyperparameters for the GNN-based policy (InforMARL) are tuned on a single task, and then k-percent tuning is followed for each different CL method.

Improvements for AI systems

Bold header: Frozen Graph Encoder (FROG) implementation

This method relies on decoupling graph processing from action prediction, specifically by using a policy where the graph encoder ψθ1 captures the structural information, while the action module µθ2 directly maps node representations to actions. This prevents forgetting in the representation learning module caused by structural changes.

Bold header: Enhanced Continual Learning Strategy

Pairing FROG with existing Continual Learning (CL) methods like Elastic Weight Consolidation (EWC) or Memory Aware Synapses (MAS) substantially improves performance on multiple CGMARL scenarios and is less prone to forgetting. This synergy leverages the generalization capacity of GNNs while confining task-specific adaptation to the action prediction module.

Bold header: Robust Hyperparameter Tuning for Plasticity

The analysis in Section D.2 suggests that the best tradeoff is achieved with λµ = 106, which is much lower than the value associated with the graph encoder (λψ = 1010). This finding supports adopting FROG to maintain plasticity, as it allows for a much lower regularization coefficient (λµ = 104) compared to standard methods, thereby enabling the model to learn new tasks without suffering from excessive forgetting.

Bold header: Structural Information Preservation in Graph-Based Policies

By freezing the graph encoder after learning the first task (freezing the graph encoder after learning the first task preserves its learned parameters), agents can preserve past structural information in graph-based CMARL policies even when adapting to new, structurally different tasks.