Modular Reinforcement Learning For Cooperative Swarms

arXiv:2605.04939 · cs.RO, cs.AI · Submitted 2026-05-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Modular Reinforcement Learning For Cooperative Swarms".

Paper summary: Rosa: So we’re looking at this paper called "Modular Reinforcement Learning For Cooperative Swarms." It tackles the huge problem of state explosion in robot swarms where robots have limited memory and computation.

Dev: Exactly, it addresses how to represent those spatial interaction states without having an exponential number of them. The core idea is a modular, decomposed representation where each feature of the state gets its own separate learning process.

Taro: So instead of one giant brain trying to figure out everything about the swarm's situation at once, you break it down into smaller parts that learn independently and then get their results fused together.

Rosa: Right, so they claim this modular approach reduces the number of states to be linear in the number of features, instead of exponential. For example, if you have k state features, there are k separate learners handling those states with n possible values for each feature.

Dev: That means instead of storing all those combinations which is exponential, you're storing something proportional to the number of features multiplied by the number of states for each feature. The paper says this leads to O(k) states total to store, which is much smaller than what a single learner would need >

Taro: But what about the actual learning part? How do these separate learners actually pick their actions and how do they agree on what to do next? That sounds like a big hurdle when you have independent processes.

Rosa: Well, each independent learner selects an action based on the state it senses, and these actions come from the entire robot's action space. If any sensor detects a neighbor within the collision radius, that triggers the learning process.

Dev: And if there’s no immediate collision threat, it responds with a fixed action, like recommending its own direction. But if there is trouble, it uses a multi-armed bandit algorithm to pick one of eight possible directions.

Taro: So they use these independent learners to handle the immediate danger responses, and then they need a way for all those different actions to combine into a single swarm decision.

Rosa: That’s where the aggregation comes in, which they call a fixed procedure nicknamed the council. This council fuses all those individual learner recommendations to generate compromises for the whole robot.

Dev: So essentially, you have these simple, independent learners handling specific parts of the spatial state, and then a fixed mechanism combines their outputs into one collective action >

Taro: So if we're thinking about real-world failure modes outside the lab where things go wrong unexpectedly, how robust is this modular setup? What happens when one of those features doesn't behave as expected?

Rosa: The experiments were done in the ARGoS3 robot swarm simulator, and they compared this modular representation against other methods like random selection or a dynamic window. They found that the modular approach performs stably, actually on par with the dynamic window method.

Paper summary: Dev: That’s interesting because it means you don't need to rely on a very complex full state learning algorithm to get good results in terms of performance, even though this modular representation is using modest memory and computation requirements >

Taro: The paper also mentioned that this modular representation can utilize rewards that cause the stateful learners to fail, which tests how gracefully the system handles unexpected inputs. How does it recover from those failures?

Rosa: They showed that when they used rewards designed to make these stateful learners fail, the modular approach still worked. Plus, they found that switching to a simpler vectorial action space actually led to improved results compared to learning collision-avoidance algorithms >

Dev: So the transition away from trying to learn collision avoidance policies seems like a pragmatic move when dealing with these constrained systems because it simplifies things and helps performance improve >

Taro: What does this mean for deployment, Rosa? If we take this out of the simulation and put it on physical robots in a messy environment, how long do you think it can run reliably before the system breaks down?

Rosa: The paper tested this in high-fidelity physical simulations, and the modular approach showed stability. The memory savings are dramatic because you’re not trying to map out every single possible interaction state exhaustively >

Dev: From a real-time loop standpoint, I'm looking at the latency introduced by that fixed action fusion mechanism. They seem to have managed to keep it manageable within their simulations, which is important for any actual deployment >

Taro: So when you think about what this means for other swarm robotics tasks, are there specific areas where this decomposition helps the most? Is it just spatial states?

Rosa: The authors suggest that modular representations are particularly well suited to spatial states because they match how potential fields work and they naturally fit with a composable action space, which is a vector of motion >

Dev: That composability is key. It means you can easily combine the outputs of different modules into a coherent movement command for the entire group, which simplifies the control interface >

Taro: So what’s their plan for taking this beyond just foraging tasks? Are they thinking about something more complex, maybe coordinating things across a much larger area or with more dynamic obstacles?

Rosa: They are intending to explore using this modular representation in other swarm robotics tasks as well, showing it has broad applicability beyond the specific foraging task used in their experiments >

Dev: It’s a solid approach because it exploits the natural structure of spatial states and action spaces without needing an enormous amount of memory or computational power from each individual robot >

Conclusion: Rosa: So, we've seen how this paper uses a modular decomposition to handle the massive number of states in robot swarms without letting memory or computation explode.

Dev: Right. The title is "Modular Reinforcement Learning For Cooperative Swarms," and it’s about using these separate learning processes that get aggregated into one plan.

Taro: Basically, they take the spatial layout of the swarm and slice it up so each piece learns what to do locally, then they combine those local decisions.

Rosa: It takes the big problem of too many states and makes the number of states linear instead of exponential based on how many sensors you have.

Dev: The authors are saying this is robust, actually performing just as well as other methods like dynamic window in their physical simulations.

Taro: And they found that even when they gave these stateful learners rewards designed to make them fail, the modular system still held up.

Rosa: It seems the big implication here is that you can get decent performance in these limited systems by using this decomposition instead of trying to map out every single interaction possibility.

Dev: It’s a significant memory saving, and they suggest this structure fits well with how physical forces and motion vectors work naturally.

Taro: But it also points toward a future where swarm coordination can be built by simply plugging in these independent modules for different parts of the spatial environment.

Bar Ilan University

cs.RO, cs.AI

Submitted: 2026-05-06

Updated: 2026-05-06

Journal ref: https://arms2026.di.unimi.it/#home

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 76/100

The gist: A modular (decomposed) representation is used for representing spatial interaction states in multi-robot reinforcement learning to overcome state explosion problems inherent in computationally…

Key concepts

Modular (Decomposed) Representation
This strategy breaks down the complex spatial state into multiple smaller parts, where each part is managed by a separate learning process. For example, spatial states are separated based on which direction a sensor is facing. This decomposition prevents the state space from growing exponentially with the number of sensors.
State Explosion Problem
This occurs when the total number of possible states in a system becomes too large to handle computationally or memory-wise. In robot swarms, explicitly representing every combination of positions and interactions between many robots leads to an unmanageable state space for traditional learning methods.
Multi-Armed Bandit Learning
This is an algorithm used by individual learners when the environment is clear. It helps the learner select one action (like a direction) from several options based on past experiences to maximize its reward, functioning as a simple decision-making tool for that specific state component.
Council Aggregation
This is the fixed procedure used to combine the individual actions selected by different learners. It fuses their preferences into a final compromise action, effectively aggregating the decisions from all modular components into one cohesive swarm movement.

Terminology

Summary

A modular (decomposed) representation is used for representing spatial interaction states in multi-robot reinforcement learning to overcome state explosion problems inherent in computationally limited robot swarms. p. 0

Where the paper says it

This paper proposes an alternative approach for representing spatial interaction states for multi-robot reinforcement learning in swarms. A modular (decomposed) representation is used, where each feature of the state is handled by a separate learning procedure, and the results aggregated.

Problem Context

Cooperative robot swarms face a significant challenge due to the limited computational resources of the robots, which prohibits explicit representation of the large combinatorial number of states (a state explosion problem). Limited memory prohibits explicit representation of the large combinatorial number of states (a state explosion problem). p. 0

Where the paper says it

A significant challenge to the use of MARL in swarm robots is raised by the limited computational resources of the robots. Limited memory prohibits explicit representation of the large combinatorial number of states (a state explosion problem).

Limited computation capabilities prohibit the use of deep neural networks as function approximations to generalize and represent states implicitly. p. 0
Where the paper says it

Limited memory prohibits explicit representation of the large combinatorial number of states (a state explosion problem). Limited computation capabilities prohibit the use of deep neural networks as function approximations to generalize and represent states implicitly.

Finally, the restrictions on communication range and bandwidth prohibit sharing of information at the swarm level. p. 0
Where the paper says it

Limited computation capabilities prohibit the use of deep neural networks as function approximations to generalize and represent states implicitly. Finally, the restrictions on communication range and bandwidth prohibit sharing of information at the swarm level.

State Representation Strategy

The paper proposes a modular (decomposed) representation where each state feature is handled by a separate learning process, and the results are aggregated. This approach reduces the number of states to be linear in the number of features, rather than exponential. p. 7

Where the paper says it

We presented a modular state representation approach that takes advantage of the natural decomposition of spatial states and the natural composability of the vectorial action space: a separate learning process is assigned to each range and bearing sensor. This leads to a total number of states that is linear in the number of sensors, rather than exponential.

For example, for k state features, there are k processes handling n possible states of the specific feature assigned to it. p. 1
Where the paper says it

This results in a state representation that grows linearly. At the extreme, a modular representation, whereby each different state feature is addressed by separate learning processes, reduces the number of states to be linear in the number of features: for 𝑘 state features, we utilize 𝑘 processes, each handling 𝑛 possible states of the specific feature assigned to it [29].

Spatial states can be decomposed by direction, assigning a separate learning process to each set of sensors facing a particular direction. p. 3
Where the paper says it

1 A Single Learner per Spatial Feature We divide up the spatial state features by direction. Specifically, for a set of sensors facing a particular direction, we assign a separate learning process.

The various states of each process are composed of the values the sensor(s) can take (jointly). p. 3
Where the paper says it

The identification of a particular process with a fixed direction means that the direction feature is implicit in the process and does not need to be explicitly represented. The various states of each process are composed of the values the sensor(s) can take (jointly).

Learning Mechanism and Aggregation

Each independent learner selects an action in response to the sensed state, and these actions are selected from the action space of the entire robot. The learning process is triggered if any sensor detects a neighbour within the collision radius. p. 3

Where the paper says it

The robot speed is fixed, and the actions choose direction. The learning process is triggered if any sensor detects a neighbour within the collision radius.

If the state is clear, it responds with a fixed action (recommending its own direction). p. 3
Where the paper says it

Then all modular learning processes are triggered. Every process responds independently as follows: if the state is clear, it responds with a fixed action (recommending its own direction).

Otherwise, it uses a multi-armed bandit learning algorithm to select one of the eight direction actions. p. 3
Where the paper says it

Every process responds independently as follows: if the state is clear, it responds with a fixed action (recommending its own direction). Otherwise, it uses a multi-armed bandit learning algorithm (Continuous UCB-1 [14]) to select one of the eight direction actions.

The recommendations are aggregated via a fixed procedure nicknamed the council, which fuses the preferences to generate compromises. p. 3
Where the paper says it

2 The Council Each of the independent learners selects an action, treated as a preference for the action to be taken by the robot. The aggregation procedure, nicknamed council, is used to fuse the preferences to generate compromises.

Experimental Validation

Experiments were conducted in high-fidelity physical simulation using the ARGoS3 robot swarm simulator. The modular representation was compared against alternatives such as random selection, dynamic window, and R-Learner. p. 5

Where the paper says it

This makes this task easier, as evident by the higher puck collection values in comparison to Arena 1 (Fig. Here, while the modular learning approach is consistently above the random selection baseline, it seems to face difficulties keeping up with dynamic window.

The results demonstrate that the modular approach performs stably, on par with dynamic window. p. 5
Where the paper says it

The modular approach performs stably, on par with dynamic window. Figure 4 shows the results from Arena 2.

Furthermore, the modular representation can utilize rewards that cause the stateful learners to fail. p. 7
Where the paper says it

The results demonstrate that the modular approach, while not always outperforming others, is robust and generally offers good performance, with dramatic memory savings. In addition, the modular representation can utilize rewards that cause the stateful learners to fail.

The transition to a simpler vectorial action space leads to improved results over learning to use collision-avoidance algorithms. p. 6
Where the paper says it

The results appear in Figures 9–11. The figure shows that learning to use vectorial actions leads to improved results over learning to use collision-avoidance algorithms to handle collisions.

Conclusion and Implications

The modular representation takes advantage of the natural decomposition of spatial states and the natural composability of the vectorial action space. This leads to a total number of states that is linear in the number of sensors, rather than exponential. p. 7

Where the paper says it

We presented a modular state representation approach that takes advantage of the natural decomposition of spatial states and the natural composability of the vectorial action space: a separate learning process is assigned to each range and bearing sensor. This leads to a total number of states that is linear in the number of sensors, rather than exponential.

The modular approach is robust and generally offers good performance, with dramatic memory savings. p. 7
Where the paper says it

We have conducted extensive experiments, contrasting the performance of the modular representation with stateful reinforcement learners. The results demonstrate that the modular approach, while not always outperforming others, is robust and generally offers good performance, with dramatic memory savings.

The authors believe modular representations are particularly well suited to spatial states because they work in similar state spaces as potential fields and naturally admit a composable action space (vectors of motion). p. 7
Where the paper says it

In addition, the modular representation can utilize rewards that cause the stateful learners to fail. We believe modular representations are particularly well suited to spatial states, as they work in similar state spaces as potential fields, and naturally admit a composable action space (vectors of motion).

The researchers intend to explore the use of this modular representation in other swarm robotics tasks as well. p. 7
Where the paper says it

Here, however, they are used as part of a learning process. We intend to explore the use of this modular representation in other swarm robotics tasks as well.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Modular state representation reduces memory requirements by decomposing spatial states into features handled by separate learning processes, moving from a combinatorial number of states to a total of O(k) states for k features instead of O(2 n) for n sensors.

  2. The improved AI system can handle more complex spatial interactions in swarm robots because the state space grows linearly with the number of features rather than exponentially with the number of entities and sensor resolution.

  3. The modular approach utilizes a vectorial action space where each sensor's learning process recommends a direction, and these recommendations are aggregated by a fixed procedure called the council to generate a single executable action.

  4. The system can effectively utilize rewards that cause stateful learners to fail, as demonstrated by the modular representation maintaining performance when switching from the difference reward Delta to the raw self-interested reward Omega.

  5. The improved AI system can be implemented on computationally limited hardware by using simple learners, such as Continuous UCB-1 algorithms for each sensor feature, instead of requiring deep neural networks for implicit state representation.

Abstract

A cooperative robot swarm is a collective of computationally-limited robots that share a common goal. Each robot can only interact with a small subset of its peers, without knowing how this affects the collective utility. Recent advances in distributed multi-agent reinforcement learning have demonstrated that it is possible for robots to learn how to interact effectively with others, in a manner that is aligned with the common goal, despite each robot learning independently of others. However, this requires each robot to represent a potentially combinatorial number of interaction states, challenging the memory capabilities of the robots. This paper proposes an alternative approach for representing spatial interaction states for multi-robot reinforcement learning in swarms. A modular (decomposed) representation is used, where each feature of the state is handled by a separate learning procedure, and the results aggregated. We demonstrate the efficacy of the approach in numerous experiments with simulated robot swarms carrying out foraging.

Sources

Related papers