HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations

summary

The gist

Multi-agent reinforcement learning has shown great promise for traffic signal control under lane-level missing observations HALight is a communication-enhanced heterogeneous multi-agent reinforcement…

In short

HALight is a communication-enhanced multi-agent reinforcement learning framework for controlling traffic signals when lane-level data is missing. It models intersections as heterogeneous systems, allowing each intersection to have unique observation and action modeling capabilities. Experiments show HALight outperforms other methods in efficiency and stability under challenging conditions like high traffic and missing observations.

Key concepts

Dec-POMDP
This framework describes the traffic signal control problem as a Decentralized Partially Observable Markov Decision Process. This means multiple agents (intersections) make decisions based on local information, but they don't see the entire state of the road network. The 'partially observable' aspect means agents only know what is directly visible to them, making coordination difficult.
Heterogeneous Graph (Ghet)
This concept models the road network where each intersection is structurally different from others. Some intersections might have different lane configurations or operational phases. This heterogeneity means that the observation and action spaces for each agent are distinct, requiring a model that can handle these diverse properties.
One-Hop Neighborhood Communication
This module allows agents to share information only with their immediate physical neighbors. Instead of broadcasting to everyone, an agent receives messages from its direct neighbors. This lightweight communication enriches the local observation by providing spatial context, helping agents coordinate their actions even when full information is unavailable.

Terminology used across episodes

This episode discusses

The paper

HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations · Read on arXiv

Zhejiang University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations".

Jane: Here's the gist of the paper. Multi-agent reinforcement learning has shown great promise for traffic signal control under lane-level missing observations HALight is a communication-enhanced heterogeneous multi-agent reinforcement learning framework for traffic signal control under lane-level missing observations This framework addresses the limitations of existing methods by supporting intersection-specific observation and action modeling and incorporating a lightweight one-hop neighborhood communication module to enrich spatial context when lane-level observations are partially unavailable Experimental results on two real-world road networks demonstrate that HALight outperforms representative homogeneous and heterogeneous baselines in terms of traffic efficiency and convergence behavior.

Tom: Problem Formulation The paper formulates the traffic signal control problem under a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) framework, defining the state space, agents, joint action space, and reward structure The formulation distinguishes between heterogeneous methods, which model the network as a directed heterogeneous graph Ghet where each intersection has distinct structural properties leading to diverse observation and action spaces Conversely, homogeneous methods model the network as a directed homogeneous graph Ghomo assuming identical structural and operation phases for all intersections The observation for heterogeneous methods is defined as the concatenation of the states of all incoming lanes at that intersection, including wait count, wave count, and current signal phase For homogeneous methods, the observation is designed to construct a unified feature representation across all intersections by aggregating lane-level features according to four directional approaches and functional lane types.

Jane: HALight Architecture HALight consists of two key components: Communication-Enhanced Heterogeneous Actors and a Centralized Critic The Communication-Enhanced Heterogeneous Actors equip each heterogeneous actor with an explicit inter-agent communication module to enhance coordination under partial observability This module restricts message exchanges to one-hop physical neighbors and integrates the received messages via a lightweight dot-product attention mechanism The local observation encoding processes the local observation together with communication information received from neighboring agents at the previous timestep Attentive Neighborhood Representation Aggregation uses a dot-product attention mechanism to selectively aggregate information from neighbors based on their relative importance, computed by the normalized attention coefficient Information Generation generates a piece of self-interested information through its communication decoder head, which is then transmitted to neighboring agents and used for decision making The agent decision process concatenates its local observation with the communication information to form a fused input vector processed by a two-layer feedforward module and then passed through a GRU Training Signal Construction HALight adopts a HAPPO-style sequential policy update scheme for heterogeneous agents, which is motivated by the need to stabilize heterogeneous agent learning The actor optimization involves computing advantage estimates Aˆ(s, a) using generalized advantage estimation (GAE) and the centralized critic with GAE To ensure consistency with joint policy improvement, HALight adopts the joint policy correction factor Mi1:m which accounts for the influence of previously updated agents on the current policy update The critic is a centralized critic that receives a global shared state integrating each agent’s local observation and communication information The critic's loss function is given by Lcritic(ϕ) = Vϕ(st) − Rˆ t squared, where the target return Rˆ t is computed using GAE Performance Comparison Extensive experiments were conducted on two real-world road networks in Hangzhou under varying demands and observation missing rates HALight outperforms representative baselines in terms of efficiency and convergence behavior across different scenarios Under full observability (pmiss = 0%), PS-DynSTGAT achieves the best ATT and AQL among the RL-based methods on the Gudang road network, while heterogeneous methods generally perform better on the more structurally diverse Xiasha road network As pmiss increases, homogeneous baselines become substantially more sensitive to missing observations and even fail to converge in several settings HALight and Het-MAPPO-RNN remain considerably more stable under increasing missing observations, suggesting that intersection-specific observation and action modeling enhances adaptation Ablation Study The ablation study evaluates the contribution of different components in HALight on the Xiasha road network under moderate and high traffic demands Results show that heterogeneous policy modeling generally achieves better performance, indicating that intersection-specific policies better accommodate structural differences among intersections Introducing one-hop neighborhood communication generally improves performance or reduces performance variation under missing observations Recurrent temporal modeling, specifically HALight-Linear, achieves lower median AWT and more concentrated distributions, especially under high demand The sequential policy update is shown to be more stable when complemented with one-hop neighborhood communication than when used without it Robustness Analysis HALight maintains relatively stable performance across all three communication perturbations—packet loss, communication delay, and communication interval—during evaluation on the Gudang road network under high demand with pmiss = 40% The results demonstrate that HALight remains robust to communication imperfections after training, with only minor performance degradation under packet loss, communication delay, and reduced communication frequency HALight maintains strong performance under direction-level missing observations on the Xiasha road network across different direction-level missing probabilities compared to PS-DynSTGAT and Het-MAPPO-RNN This demonstrates that HALight handles correlated missing observations across multiple lanes of the same direction, showing adaptability to more structured missing patterns The communication efficiency analysis shows that the overall communication related computation scales linearly with both n and E, suggesting HALight can be extended to large-scale road networks In this paper, HALight is proposed as a communication-enhanced heterogeneous MARL framework that supports intersection-specific observation/action modeling and integrates lightweight one-hop neighborhood communication with temporal modeling The framework demonstrates effectiveness across two real-world road networks, covering different network scales, traffic demands, and observation missing rates HALight maintains comparable throughput while retaining advantages in other evaluated metrics under challenging settings like high demand and severe observation missingness The proposed communication design scales favorably with network size, indicating its potential for large-scale deployment.

Tom: References

3: P.

Paper summary: Lu: So we've covered the paper, "HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations". We looked at how it handles structural differences, communication, and missing data >

Jane: It seems the big idea is that by moving away from assuming everything is identical and adding targeted communication between neighbors, we can build a system that learns to adapt to messy real world traffic situations >

Tom: It’s about taking the complexity of different intersection types—the heterogeneity—and making it manageable by giving each agent specific tools to observe its environment and talk to its immediate neighbors >

Meng: From a practical side, if this works on those two road networks, it suggests that we can build control systems that are more robust even when sensors aren't perfect or when the traffic patterns change unexpectedly >

Lalam: I think what really matters is the scaling analysis they did. The communication related computation scales linearly with both n and E, which means HALight can actually be extended to large scale road networks >

Lu: That linear scaling is a big deal for deployment because it means the complexity doesn't explode as you add more intersections or lanes >

Jane: So, in simple terms, HALight offers a way to manage the mess of real city traffic control by letting each intersection figure out its own specific rules and coordinating locally with its immediate neighbors >

Tom: It’s not about one magic solution, it’s about adding targeted communication and structural awareness to make learning more stable under imperfect conditions >

Conclusion: Jane: So we’ve seen how HALight tackles traffic control when you don't have perfect information about individual lanes, focusing on making the system smarter by letting each intersection learn its own way to act >

Tom: Yeah, the authors are showing this framework handles intersections that are structurally different from each other, which is key because real cities aren't uniform >

Lu: Exactly. They set up a problem where some intersections have different layouts and others don't, and they use a communication method to help them coordinate without knowing everything >

Meng: But what’s the actual payoff for someone trying to deploy this? Is it just theoretically sound, or can it actually handle the chaos of real-world traffic flow >

Lalam: It suggests that instead of forcing every intersection to follow one single set of rules, you let them specialize and talk to their neighbors >

Jane: So if we look at the title, "HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations," what does that actually mean for a driver on the road? >

Tom: It means this system is designed to manage traffic signals in a city where you can't see every single lane perfectly and each intersection acts differently >

Lu: The paper shows they tested it on two different real-world road networks, which means the idea isn't just theoretical; it’s been put into practice on actual data >

Meng: Okay, so when they ran those tests under high traffic demands, did the system stay stable or did it start failing because of the missing observations >

Lalam: It showed that HALight maintained a lot of its performance even when about forty percent of the lane-level information was missing, which is pretty solid >

Jane: That’s important for us listening. It means this framework adapts well to real-world messiness instead of just breaking down when things get complicated >

Tom: The authors conclude that by combining intersection-specific modeling with a simple communication link, you get a system that works better than the standard baselines, even when traffic gets really heavy >

Lu: It’s about finding that balance between letting agents be specialized and giving them enough context to cooperate across the network >

Meng: I'm still thinking about the cost of running this at a massive scale, but it seems like their communication scaling analysis is good, showing it doesn't get bogged down by the sheer number of intersections >

Lalam: It scales linearly with both the number of agents and how many connections there are, which means it could handle huge road networks without needing a super heavy computational setup >

Jane: So, to wrap up this part, HALight offers a way to control city traffic when you have incomplete data by making intersections smarter and letting them communicate locally >

Tom: It’s about moving from one big, dumb control system to many smaller, specialized ones that can actually adapt when things get messy >

Lu: This suggests the future direction is really about building these communication layers so they can handle even more complex patterns of missing data >

More episodes

← Home