Mitigating Bus Bunching with Reinforcement Learning Enhanced by Semantic Stop Embedding

This paper introduces an LLM-assisted semantic stop representation for reinforcement-learning-based bus holding control. The study addresses bus bunching, a persistent problem in high-frequency transit where small disturbances can disrupt vehicle spacing and trigger a self-reinforcing process. The authors note that existing reinforcement-learning-based holding controllers "primarily rely on instantaneous operational variables or route-specific stop identifiers, which provide limited information about the functional and operational context of individual stops and constrain policy reuse across routes."

The proposed approach uses an LLM offline to transform heterogeneous stop information—including physical attributes, surrounding activity context, and historical operational characteristics—into fixed semantic embeddings that are incorporated into a deep Q-learning controller without requiring real-time LLM inference. The semantic representation is constructed through a two-stage process: first, a deterministic prompt-construction function converts structured stop records into constrained prompts, and an LLM maps these to structured semantic metadata including functional role tags, demand-related tags, and data-quality indicators; second, a pretrained text-embedding model maps the canonical text representation to a high-dimensional vector, which is then standardized and projected into a compact 8-dimensional feature space using PCA.

The framework is formulated as an event-driven, partially observed Markov decision process where holding decisions are made asynchronously at bus-departure events. The observation combines normalized forward and backward spatial spacings with the fixed semantic embedding of the current stop. A shared DQN selects discrete holding durations from an action set of 0, 20, 40, and 60 seconds. The reward function combines a spacing-equalization term, a penalty for newly detected bunching conditions, and a regularization term favoring shorter holding durations.

Experiments were conducted in stochastic simulations calibrated with observed data from two Pennsylvania State University campus shuttle routes: the White Loop (WL) and Blue Loop (BL). The simulation environment uses historical CATA automatic vehicle location records from August 15 to December 15, 2017, to estimate stop-specific Poisson passenger-arrival rates, gravity-model alighting probabilities, and link-specific lognormal travel-time distributions.

The within-route results show that compared with the best calibrated Daganzo baseline, the semantic controller reduces headway variability, bunching events, and passenger waiting time by 32.0%, 69.2%, and 24.0%, respectively. Specifically, the semantic policy achieves the lowest mean headway variation (128.7 ± 21.4 s), passenger waiting time (2.57 ± 0.3 min), and holding time (25.1 ± 1.0 s) among the RL controllers, while attaining the second-lowest bunching count (9.7 ± 3.8). The paper notes that "a route-specific stop identifier does not improve the spacing-only controller, whereas semantic stop information improves headway regularity, waiting time, and holding effort, providing a more favorable overall trade-off across control objectives."

The cross-route experiments use WL as the source route and BL as the target route, examining three settings: zero-shot transfer, warm-start fine-tuning, and cold-start training. The results show that "zero-shot transfer provides limited immediate generalization, while warm-start fine-tuning accelerates early-stage learning and improves transferred policies; cold-start training nevertheless achieves the best final performance." The warm-start policy begins from better initial performance with higher total reward and lower headway variation during early training episodes, reaching a stable performance range more quickly. Cold-start training achieves the lowest headway variation (201.2 ± 68.0 s), bunching count (18.2 ± 12.3), and passenger waiting time (3.79 ± 1.0 min) among RL variants, but with greater holding effort (22.4 ± 3.3 s) than transferred policies.

The paper also examines stop-level holding patterns, finding that the nonuniform profiles show that both policies learn stop-specific control patterns rather than applying holding uniformly along the route. Adding semantic stop information changes where holding is applied: the unweighted stop-level average hold probability decreases from 0.612 to 0.508 on WL and from 0.582 to 0.561 on BL, while the locations with increased or reduced holding also shift. The authors suggest that semantic stop information affects the spatial allocation of control effort and may help the policy identify more suitable locations for regulating downstream service.

The authors conclude that semantic stop representations can complement conventional bus holding states and support source-to-target policy reuse and adaptation in the studied setting. From an implementation perspective, the representations are constructed and cached offline, so language-model inference, text embedding, and representation alignment do not add computation or latency to the real-time control loop. The study's limitations include being limited to simulation-based experiments on two university shuttle routes and one primary transfer direction, with future work needed to evaluate the framework on larger networks, quantify data and computational savings, examine additional transfer directions, and test robustness under demand shifts and disruptions.

Improvements to AI Systems:

1. Offline LLM-Enhanced State Representation for Real-Time Control: Integrate a pre-processing pipeline where an LLM converts heterogeneous, unstructured environmental data (e.g., stop attributes, surrounding context, historical patterns) into fixed, low-dimensional semantic embeddings (via text embedding + PCA) that are cached. This allows real-time RL controllers to leverage rich semantic context without incurring LLM inference latency or computational overhead during execution.

2. Transferable Policy Initialization via Semantic Embeddings: Use semantic stop embeddings as a shared feature space to enable cross-route policy reuse. The improved system can warm-start RL training on a new route by initializing the Q-network with weights from a source route, using the semantic embeddings to align state spaces, thereby accelerating early-stage learning and reducing sample complexity compared to cold-start or route-specific identifiers.

3. Context-Aware Control Effort Allocation: Enhance the RL agent’s observation space with semantic stop features (functional role, demand tags, data quality) so the policy learns non-uniform, stop-specific control actions (e.g., holding durations) that are spatially optimized. The improved system can dynamically allocate control effort to stops where it is most effective for downstream regularity, rather than applying uniform or route-agnostic actions.

4. Robustness to Route Heterogeneity: Replace brittle, route-specific stop identifiers with generalizable semantic embeddings. The improved system can transfer knowledge across different transit routes or networks with distinct stop characteristics, reducing the need for retraining from scratch and improving zero-shot or few-shot generalization in new operational contexts.

5. Event-Driven Asynchronous Decision-Making with Semantic Context: Build an event-driven POMDP controller that makes decisions at departure events, using a combined observation of real-time spacings and cached semantic embeddings. The improved system can handle asynchronous, partially observed environments while maintaining semantic awareness, leading to better headway regularity, reduced bunching, and lower passenger waiting times.

What the Improved AI System Can Do:

• Operate real-time bus holding control with zero added latency, using precomputed semantic stop representations to make smarter, context-aware decisions.

• Transfer control policies across different bus routes with warm-start fine-tuning, achieving faster convergence and better initial performance than training from scratch, while maintaining competitive final performance.

• Reduce headway variability by up to 32%, bunching events by up to 69%, and passenger waiting time by up to 24% compared to non-semantic baselines, as demonstrated in simulation.

• Learn stop-specific holding patterns that spatially allocate control effort more effectively, reducing average hold probability and targeting locations that improve downstream service.

• Adapt to new routes with limited data by leveraging semantic embeddings to align state spaces, enabling quicker deployment in transit networks with different stop characteristics.