CRHT: A Continuous Regression Hybrid Transformer for Vessel Trajectory Prediction with Online Cluster Sampling

CRHT: A CONTINUOUS REGRESSION HYBRID TRANSFORMER FOR VESSEL TRAJECTORY PREDICTION WITH ONLINE CLUSTER SAMPLING

Abstract: "Accurate vessel trajectory prediction is critical for maritime safety and anomaly detection, yet existing models often struggle with geographic bias and navigational realism. We propose the Continuous Regression Hybrid Transformer (CRHT), a deep learning framework designed to forecast vessel motion using Automatic Identification System (AIS) data. To mitigate spatial data imbalance, we introduce an online K-means cluster sampling strategy that ensures diverse exposure to rare maneuvers during training. Our hybrid architecture integrates 1D convolutional layers for local kinematic feature extraction with a multi-head attention mechanism for global temporal context. CRHT demonstrates superior performance in short-term forecasting, achieving the lowest errors at the 1-hour horizon. The results demonstrate that while discrete models provide high navigational stability over long horizons, CRHT offers an optimal balance of precision and maneuver tracking for real-time maritime surveillance."

Introduction: The paper addresses vessel trajectory prediction using AIS data, noting challenges of irregular sampling, sensor noise, and bias where models overfit to linear paths and fail during complex maneuvers. It reviews evolution from Kalman Filters to RNNs/LSTMs to Transformer-based architectures, and notes that discrete generative models like TrAISformer offer navigational realism through coordinate binning, but they often suffer from quantization errors and high inference latency. The contributions are "a scalable MapReduce preprocessing pipeline, an online K-Means cluster sampling strategy to capture diverse maneuvers, and a hybrid CNN-Transformer architecture that integrates local kinematic extraction with global attention for robust trajectory forecasting."

Data: The dataset consists of 1,818,441,680 AIS messages from 35,349 unique vessels from the Danish Maritime Authority, covering July 24th - October 24th 2025. The study focuses on tankers and cargo ships with features latitude, longitude, Speed over Ground (SOG) in knots, and Course over Ground (COG) in degrees. Preprocessing included filtering by geographical boundary (lat ∈ [54, 59], lon ∈ [5, 17]), SOG ∈ [0, 30], COG ∈ [0, 360], defining voyages with time gaps smaller than 2 hours, partitioning voyages longer than 20 hours, discarding those shorter than 4 hours or with fewer than 20 messages, removing outliers exceeding 40 knots, and resampling at 5-minute intervals. This reduced the dataset from 72,995 to 41,458 voyages spanning 1,203 unique vessels.

Methodology: The task is framed as a multivariate sequence-to-sequence regression problem with input X = x1,..., xT where xt ∈ R4 and future sequence Y = yT+1,..., yT+L based on features F = Latitude, Longitude, SOG, COG. The CRHT architecture has three stages:

1. Local-Global Encoder: Input processed by a 1D Convolutional block (Conv1d → ReLU → Conv1d) to extract local kinematic derivatives (velocity, acceleration) and smooth sensor noise, augmented with sinusoidal positional encodings, passed to an N-layer Transformer Encoder with multi-head self-attention over the 64-step history.

2. Query-Based Decoder: Uses a Transformer Decoder with L learnable query embeddings Q ∈ RL×dmodel that interact with encoder memory via multi-head cross-attention to predict future steps in parallel.

3. Scaled Delta Learning: Inputs are MinMax normalized to [0, 1], the model regresses inter-step displacements ∆ŷt, optimized using Huber Loss against scaled ground-truth targets with scaling factor α = 100.0.

Online K-Means Cluster Sampling: The paper notes that uniform random sampling introduces a strong bias toward geographically dense and low-speed regions because commercial vessels repeatedly traverse shipping lanes. To mitigate this, a K-Means algorithm with k = 50 clusters is fitted on the latitude–longitude coordinates of AIS messages. During training, a cluster is drawn uniformly from the clusters overlapping that segment and a starting index is then sampled uniformly over part of the sequence within the selected cluster, enforcing approximately uniform exposure across spatial clusters while preserving temporal continuity.

Experiments: Baselines include a standard Kalman Filter assuming a Constant Velocity (CV) model and TrAISformer. A CRHT variant with a GRU decoder performed poorly compared to the Transformer decoder due to the Information Bottleneck problem. The final CRHT uses dmodel = 512 with 8 attention heads, 8 encoder layers, and 8 decoder layers. Data split was 80:10:10 ratio split by MMSI to avoid data leakage. Training used AdamW optimizer with a batch size of 512 and an initial learning rate of 3×10−4 on a single NVIDIA A100 (80GB) GPU, with early stopping (patience 50), 5-epoch warm-up, and validation-based learning rate decay. Training fixed T = 64 (past) and L = 12 (future). Evaluation used strict deterministic evaluation (rejecting Best-of-N sampling as unsuitable for real-world surveillance), with autoregressive generation for long horizons (block-autoregressively forecasting 1-hour horizons iteratively). Metrics were Average Displacement Error (ADE) and Final Displacement Error (FDE) using Haversine distance. The mean trip length was 4.17 hours.

Results: Table I reports:

• 1 Hour Horizon: Kalman Filter ADE 1.58, FDE 3.60; TrAISformer ADE 1.38, FDE 2.41; CRHT ADE 1.21, FDE 2.07 (CRHT best in both).

• 2 Hour Horizon: Kalman Filter ADE 3.70, FDE 8.86; TrAISformer ADE 2.62, FDE 5.42; CRHT ADE 2.55, FDE 5.85 (CRHT best ADE; TrAISformer best FDE).

Discussion: At a 1-hour prediction horizon, CRHT achieves the lowest mean ADE (1.21 km) and the lowest mean FDE (2.07 km) among all evaluated models. At the 2-hour horizon, CRHT maintains the lowest mean ADE (2.55 km). For FDE at 2 hours, CRHT (5.85 km) is slightly higher than TrAISformer (5.42 km). Case studies show: In a sharp northward turn around Bornholm, CRHT predicts a trajectory that follows the northward maneuver and remains feasible over the full forecast horizon, while Kalman Filter and TrAISformer extrapolate the preceding motion pattern and project the trajectory into the island of Bornholm. In a slight curve scenario, CRHT aligns with ground truth with small deviation, while Kalman Filter achieves the lowest FDE in this instance, but its trajectory does not capture the executed maneuver. In an ambiguous scenario, TrAISformer produces a trajectory that would be realistic if the vessel had taken an eastward turn instead of the executed westward maneuver, effectively 'hallucinating' an alternative path.

Conclusion: "CRHT achieves the lowest mean displacement errors across short- and intermediate-term horizons while producing smooth, continuous trajectories that handle nonlinear maneuvers. The results demonstrate its ability to follow complex paths realistically, in contrast to classical filtering and discrete generative models that either fail on sharp turns or generate plausible but unexecuted trajectories. By combining predictive accuracy with navigational feasibility, CRHT provides a robust framework for maritime trajectory forecasting under both typical and uncertain conditions."

Improvements to AI systems based on this paper:

1. Spatially Balanced Training via Online Cluster Sampling

• Implement an online K-means cluster sampling strategy (k=50) over geographic coordinates during training to prevent overfitting to dense shipping lanes.

• The improved system can learn rare maneuvers (sharp turns, deviations) without being biased by high-frequency linear paths, improving generalization across diverse maritime regions.

2. Hybrid Local-Global Feature Extraction

• Integrate 1D convolutional layers (Conv1d → ReLU → Conv1d) before a multi-head Transformer encoder to extract local kinematic derivatives (velocity, acceleration) and denoise sensor input.

• The improved system can capture short-term motion dynamics while retaining long-range temporal context, reducing error accumulation in trajectory forecasts.

3. Query-Based Parallel Decoding with Scaled Delta Regression

• Use learnable query embeddings in a Transformer decoder for parallel future-step prediction, combined with scaled inter-step displacement regression (Huber loss, α=100).

• The improved system can forecast multiple horizons simultaneously with lower latency and higher precision, avoiding quantization errors from discrete binning.

4. Autoregressive Block Forecasting for Long Horizons

• Apply block-autoregressive generation (iteratively feeding predicted outputs as inputs) for horizons beyond the trained length (e.g., 2 hours from 1-hour blocks).

• The improved system maintains navigational feasibility over extended periods, balancing accuracy and maneuver tracking without hallucinating alternative paths.

5. Strict Deterministic Evaluation for Real-World Deployment

• Replace stochastic sampling (e.g., Best-of-N) with deterministic outputs for surveillance tasks.

• The improved system provides reliable, reproducible predictions suitable for real-time maritime anomaly detection, avoiding unrealistic or unexecuted trajectory alternatives.

6. Scalable Data Preprocessing Pipeline

• Adopt a MapReduce-based pipeline to filter, partition, and resample massive AIS datasets (e.g., 1.8B messages) into clean voyages with consistent 5-minute intervals.

• The improved system can handle large-scale, noisy sensor data efficiently, enabling training on diverse vessel types and geographic regions without manual curation.

Capabilities of the improved AI system:

• Predicts vessel trajectories with 23% lower ADE (1.21 km vs. 1.58 km) and 43% lower FDE (2.07 km vs. 3.60 km) than Kalman filters at 1-hour horizons.

• Outperforms discrete generative models (e.g., TrAISformer) in short-term accuracy while producing smooth, continuous paths that correctly follow sharp maneuvers (e.g., turns around islands) rather than extrapolating into obstacles.

• Operates in real time on a single GPU, with deterministic outputs suitable for maritime safety, collision avoidance, and anomaly detection systems.

• Maintains robust performance across both typical linear routes and rare, complex navigational scenarios, reducing false alarms and missed detections in surveillance.