Reinforcement Learning based Traffic Signal Control for Queue Minimization: From Lane-based to Laneless Traffic
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Reinforcement Learning based Traffic Signal Control for Queue Minimization".
Dev: Here's the gist of the paper. Efficient traffic signal control (TSC) is crucial for reducing congestion, travel delays, pollution, and for ensuring road safety <ref:2509.21745#pg2> The proposed algorithm has been implemented using the Simulation of Urban Mobility (SUMO) traffic simulator and demonstrates superior performance over both traditional methods and other conventional RL-based approaches in reducing queue lengths <ref:2509.21745#pg7>.
Rosa: Problem Formulation The TSC problem is formulated as a Markov Decision Process (MDP), where at each discrete time step t, the controller observes the current traffic state and selects an action to optimize signal timings <ref:2509.21745#pg4>.
Dev: The state space in TSC provides a detailed representation of the traffic in the signalized intersection such as the current cycle time, the active signal phase, queue lengths, and remaining phase time <ref:2509.21745#pg5>.
Rosa: The action space is defined as at ∈ 0, 1, 2 where each action corresponds to: 0: End the current phase immediately; 1: Continue the current phase with no change (i.e., maintain green time for another δtime seconds); and 2: Extend the current phase by an additional δtime seconds <ref:2509.21745#pg9>.
Paper summary: Dev: So looking at the paper Reinforcement Learning based Traffic Signal Control for Queue Minimization From Lane-based to Laneless Traffic, the authors are proposing a framework that uses reinforcement learning with ppo to minimize queue lengths by moving beyond lane-based control.
Rosa: They claim their method is superior in reducing queue lengths compared to traditional approaches and other conventional reinforcement learning methods when tested in the sumo traffic simulator.
Taro: The implication for real-world systems is that this kind of adaptive control could lead to better coordination along traffic corridors because it accounts for long-term and downstream effects, which is something actuation control often misses.
Dev: They focused on making the state representation robust by using different methods like an autoencoder, and they found that the sixteen-D latent space autoencoder representation performed the best in minimizing total queue length.
Rosa: The authors also showed that their green time allocation becomes adaptive and directly correlates with the observed queue length during each cycle, which is a key finding for understanding how to build this policy.
Taro: What this means in simpler terms is that instead of just reacting to what's happening right now at one signal, the system learns a strategy that manages the overall traffic flow better by adjusting timings based on the queues across all approaches.
Dev: The title itself, moving from lane-based to laneless control, suggests they are aiming for a more comprehensive optimization across the entire intersection network rather than just managing individual lanes in isolation.
Rosa: It’s about using reinforcement learning to find those better, coordinated timings that reduce congestion and travel delays overall.
Conclusion: Rosa: So we're wrapping up this look at the paper "Reinforcement Learning based Traffic Signal Control for Queue Minimization: From Lane-based to Laneless Traffic." Basically, they took a traffic signal and used reinforcement learning to learn how to manage it better than the old ways.
Dev: Yeah, it moves beyond just controlling individual lanes and tries to optimize the whole intersection at once. The authors say this approach is better at cutting down on queues in the simulation compared to traditional methods.
Taro: It seems like they were really focused on making sure this wasn't just a simple fix but something that actually learns how traffic flows together across different directions. I'm interested in whether that learning translates when you put it outside the lab.
Rosa: That’s what I want to know too. The paper shows how they used a specific way to describe the traffic—a sixteen-dimensional space—which they found helped them get the best results for minimizing total queues. But does that just work in practice when the real world gets messy?
Dev: The simulation results are solid, but we have to ask about latency and how fast this whole control loop runs. If the decision takes too long, you’ve got a different kind of problem entirely, right? That's where the safety case comes in for me.
Taro: Exactly. It's not just about getting a lower number on paper; it’s about what happens when the unexpected stuff goes down and how gracefully the system recovers without causing a bigger mess than it started with.
Rosa: So, while this paper shows some pretty cool learning results in controlled environments, we still need to see if these kinds of adaptive strategies hold up under real-world conditions where traffic isn't perfectly predictable.
Dev: We’ll have to dig into those limitations next. That’s what I mean—where does this model stop making sense when the world misbehaves?
Department of Civil Engineering, IIT Madras · Wadhwani School of Data Science & AI, IIT Madras · School of Electrical Engineering & Robotics, QUT, Brisbane, Australia · Robert Bosch Centre for Data Science & AI, IIT Madras
eess.SY, cs.LG, cs.SY
Submitted: 2025-09-26
Updated: 2026-11-03
Code: https://github.com/AnirudN/RLTSCQ
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 74/100
The gist: Efficient traffic signal control (TSC) is crucial for reducing congestion, travel delays, pollution, and for ensuring road safety <ref:2509.21745#pg2> The proposed algorithm has been implemented…
Key concepts
- Markov Decision Process (MDP)
- The problem is framed as an MDP where a controller observes the current traffic state and chooses an action to optimize signal timings. The 'state' includes details like cycle time and queue lengths, while the 'action' determines how to adjust the signal phase.
- State Space Representation
- Researchers explored various ways to describe traffic conditions for the RL agent. They tested different representations, including preliminary and expanded states, learned representations via an autoencoder, and a K-Planes inspired transformation, seeking one that captures key traffic information efficiently and robustly.
- PPO Algorithm
- The Proximal Policy Optimization (PPO) algorithm is the RL framework used for training the traffic controller. It optimizes a clipped surrogate objective to encourage policy improvement while ensuring that updates to the signal timing strategy are not too drastic, leading to stable learning.
- Queue-Based Reward Signal
- Instead of using delay as a reward, this method uses total queue length as the reward signal. This approach proved highly effective, showing that assigning longer green times to phases with higher queues directly correlates with minimizing the overall queue length across all approaches.
Terminology
Summary
Efficient traffic signal control (TSC) is crucial for reducing congestion, travel delays, pollution, and for ensuring road safety <ref:2509. p. 0 JOURNAL OF LATEX CLASS FILES, VOL. 9, SEPTEMBER 2020 1
Reinforcement Learning Based Traffic Signal Design to Minimize
Queue Lengths
Anirud Nandakumar1,2
, Chayan Banerjee3,∗
, Lelitha Devi Vanajakshi1,4
Abstract—Efficient traffic signal control (TSC) is crucial for
reducing congestion, travel delays, pollution, and for ensuring
road safety. The challenge of
efficiently representing highly stochastic traffic conditions for
an RL controller is addressed through multiple state representations, including an expanded state space, an autoencoder
representation, and a K-Planes-inspired representation. The proposed algorithm has been implemented using the Simulation
of Urban Mobility (SUMO) traffic simulator and demonstrates
superior performance over both traditional methods and other
conventional RL-based approaches in reducing queue lengths.Where the paper says it
Where the paper says it
Problem Formulation
The TSC problem is formulated as a Markov Decision Process (MDP), where at each discrete time step t, the controller observes the current traffic state and selects an action to optimize signal timings <ref:2509.21745#pg4>. The state space in TSC provides a detailed representation of the traffic in the signalized intersection such as the current cycle time, the active signal phase, queue lengths, and remaining phase time <ref:2509. p. 1 , to change the green time of the current phase in
order to minimize congestion and maximize efficiency. The
state space in TSC provides a detailed representation of the
traffic in the signalized intersection such as the current cycle
time, the active signal phase, the duration of green lights for
each phase and the queue lengths in each lane. The action space is defined as:
at ∈ {0, 1, 2},
where each action corresponds to:
• 0: End the current phase immediately. • 1: Continue the current phase with no change (i. ,
maintain green time for another δtime seconds). • 2: Extend the current phase by an additional δtime seconds.Where the paper says it
Where the paper says it
Where the paper says it
State Representation Designs
Different state space representations were explored with the aim of achieving a more data-efficient and noise-robust abstraction of the environment <ref:2509.21745#pg7>. The paper introduced several state space representations for efficiently capturing key traffic information, including a preliminary state space, an expanded state space, learned representations using an autoencoder, and a K-Planes inspired fixed transformation <ref:2509. p. 2 This feedback is used to update the policy, progressively improving traffic flow. introduces several state space representations for efficiently
capturing key traffic information, including a preliminary state
space, an expanded state space, learned representations using
an autoencoder, and a K-Planes inspired fixed transformation,
all of which are detailed in section V-A. An analysis of the performance of various autoencoder representations is presented in Table IV. A 16-D latent space
autoencoder demonstrated the best performance by minimizing
total queue length.Where the paper says it
Where the paper says it
How the training signal is built
The overall methodology involves setting up an RL framework that interacts with a SUMO traffic simulator <ref:2509.21745#pg7>. The PPO algorithm optimizes a clipped surrogate objective that encourages improvement while limiting the update step size <ref:2509. p. 4 PPO Algorithm
In this study, PPO[12] is used as the RL controller for
adaptive traffic signal control. PPO optimizes a clipped surrogate objective that encourages improvement while limiting
the update step size:
L
CLIP(θ) = Et
min rt(θ)Aˆ
t, clip(rt(θ), 1 − ϵ, 1 + ϵ)Aˆ
t
,
(14)
where
rt(θ) = πθ(at|st)
πθold (at|st)
, (15)
is the probability ratio between new and old policies, Aˆ
t is
the advantage estimate and ϵ is a small hyperparameter that
controls the extent of the policy update. PPO has demonstrated strong empirical performance and is
widely used in continuous control tasks such as robotic manipulation and traffic signal optimization. Training Framework
Our training framework is an iterative interaction loop
between the PPO-based Traffic Controller and the SUMO
simulation environment, as illustrated in Fig.Where the paper says it
Where the paper says it
Where the gains come from
The queue-based RL algorithm with 16-D autoencoder representation surpasses all the other baselines, including delay-based reward formulation <ref:2509.21745#pg9>. The results highlight the effectiveness of queue length as a reward signal for learning efficient TSC policies <ref:2509. p. 9 Furthermore, the
PPO controller trained with a queue length-based reward also
surpasses other RL controllers trained with alternative reward
formulations, including those optimized for minimizing delay,
maximizing average speed, and minimizing pressure. The
results highlight the effectiveness of queue length as a reward
signal for learning efficient TSC policies. The
results highlight the effectiveness of queue length as a reward
signal for learning efficient TSC policies. Phase-level analysis
revealed that the green time allocation becomes adaptive and
correlates with the queue length observed during each cycle;
phases with higher queues were assigned longer green times. For
each approach j, let Lj be the set of lanes belonging to
approach j, and for each lane l ∈ Lj , let q
max
l
represent the
maximum queue length observed on lane l during the cycle. The total queue length for approach j is computed as the
maximum queue length across its lanes:
Q
approach
j = max
l∈Lj
q
max
l
, (16)
where Q
approach
j
is the maximum queue observed in the entire
approach j during the cycle. The total queue length for approach j is computed as the
maximum queue length across its lanes:
Q
approach
j = max
l∈Lj
q
max
l
, (16)
where Q
approach
j
is the maximum queue observed in the entire
approach j during the cycle. The total queue length for the
entire cycle is then computed as the sum of maximum queue
lengths from all approaches:
Qcycle =
X
N
j=1
Q
approach
j
, (17)
where Qcycle represents the total queue length for the cycle
based on per-approach maximum queues.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
The RL controller can dynamically adjust green light timings based on observed queue lengths to minimize total queue lengths across all phases, directly optimizing for a macroscopic traffic objective rather than individual vehicle delays.
-
The system can utilize novel state space representations like the 16-D autoencoder representation, which is shown to demonstrate
superior performance
in minimizing total queue length compared to other state abstractions. -
The AI system can be trained using the Proximal Policy Optimization (PPO) algorithm, which is noted for its
stable learning and sampling efficiency,
making it well-suited for continuous control tasks like traffic signal optimization. -
The controller can be compared against traditional methods like the Dynamic Webster Method, achieving an
approximately 29% reduction in average queue lengths compared to the traditional Webster method
in simulation. -
The RL framework can be extended to incorporate alternative reward formulations, such as delay-based rewards or pressure-based rewards, allowing the system to learn policies optimized for minimizing total accumulated delay or balancing inflow/outflow pressures.
Abstract
Efficient traffic signal control (TSC) is essential for reducing congestion and delays. Traditional TSC methods struggle to capture the dynamics of laneless, heterogeneous traffic. We propose a reinforcement learning (RL) based adaptive TSC framework whose reward depends only on approach-level queue length, motivated by the difficulty of obtaining per-vehicle measurements in laneless traffic, and evaluate it using SUMO on a conventional lane-based, homogeneous (LBHO) intersection and on a non-orthogonal laneless, heterogeneous (LLHE) intersection. On the LBHO site, the lowest mean configuration has about 23% lower mean queue length than a Dynamic Webster baseline, and the unselected Full-State configuration about 15% lower. On the LLHE site, all RL configurations have 66-73% lower mean queue length than a Dynamic Webster baseline under nominal demand, and 63--74% lower under two higher-demand scenarios evaluated without retraining. Our algorithm also outperforms actuated control and a RESCO-DQN baseline on mean queue length. We additionally release a laneless, heterogeneous-vehicle SUMO framework integrated with RL based signal control, to our knowledge the first such open implementation on a current SUMO version.
Sources
- Implementing the Deep Q-Network
- Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
- Proximal Policy Optimization Algorithms
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction