CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving

CRUISE is a novel uncertainty-aware cross-modal sensor fusion framework for robust autonomous driving. The paper states: "Modern autonomous vehicles are equipped with multiple sensors, such as cameras, LiDAR, and radar, for comprehensive environmental perception. However, robust cross-modal feature fusion remains a critical challenge, as the reliability of each sensor varies significantly across diverse real-world driving conditions, including poor visibility and adverse weather."

The paper identifies a key limitation in existing methods: existing uncertainty-aware fusion methods typically rely on simple feature-level uncertainty estimates and thus often fail to generalize effectively in complex, out-of-distribution scenarios. Additionally, "most existing methods rely on coarse task-level or feature-level UQ, assigning a single uncertainty score to an entire modality or high-level representation. This results in an insufficient guidance signal. Consequently, they fail to capture the necessary fine-grained uncertainty variations at the pixel or voxel level, which are essential for dense prediction tasks like semantic segmentation and accurate 3D detection. Furthermore, these methods typically process each modality independently, often treating them as isolated data streams and lacking mechanisms to explicitly capture cross-modal dependencies."

To address these limitations, CRUISE integrates two key innovations. First, the paper proposes "a VLM-guided UQ module that generates fine-grained, pixel-level uncertainty estimates. By leveraging the VLM's rich prior knowledge and superior contextual reasoning, our approach provides a highly informative guide for the fusion process. Specifically, the authors adapt a pre-trained VLM (Anomaly GPT), originally developed for industrial anomaly detection, to the autonomous driving domain via LoRA fine-tuning on AV datasets. Second, the paper introduces a dynamic adaptation mechanism that explicitly models and captures cross-modal dependencies, ensuring the framework fully exploits the inherent complementary nature of multi-sensor inputs."

The methodology is structured into two main stages: VLM-guided UQ and uncertainty-aware fusion. In the first stage, CRUISE first projects features captured by different modalities into a unified shared feature space, typically a Bird's-Eye-View (BEV) representation, to ensure feature alignment. A prompt adapter is fine-tuned for the VLM, and during training, we construct a feature pool for each modality, storing features of representative training samples as a reference for uncertainty estimation. During inference, "we compute an initial uncertainty estimate by comparing the test sample's feature map with these representative training samples in the feature pool. The VLM then refines these estimates through global contextual reasoning, leveraging its deep semantic understanding to generate a highly informative uncertainty heatmap that accurately localizes unreliable regions at a fine-grained spatial level."

The VLM-guided UQ module has a visual stream and a textual stream. The visual stream "processes input feature representations... using a frozen ImageBind encoder, which extracts multi-scale patch-level features from hierarchical layers... These features are then passed through a lightweight transformer-based decoder that aligns visual features with learned textual semantics... The decoder computes similarity scores between patch-level features and textual features, then applies up-sampling to generate a high-resolution localization map M. The textual stream interprets uncertainty via attention-based reasoning using four inputs: (i) a localization map M from the visual stream, (ii) a user query, (iii) a structured prompt from the prompt adapter, and (iv) feature representations... generating an attention-based uncertainty map Atext. The final uncertainty heatmap is computed as Wrm = λM + γAtext, where λ and γ are hyperparameters that control the relative contributions of the visual and textual information."

In the uncertainty-aware fusion stage, we first use the uncertainty heatmap Wrm as a mask to suppress unreliable regions, generating a refined representation Xi′ = Xi ⊙ (1 − Wrm). The fusion module employs a combination of spatial, channel, and pixel-level attention mechanisms to capture comprehensive contextual information. Spatial attention captures spatial dependencies by generating a weight map Si for each modality's feature map Xi′, channel attention enhances informative channels by dynamically adjusting their importance, and pixel attention facilitates cross-modal fusion by adaptively weighting each modality at the pixel level.

The dynamic adaptation mechanism adaptively adjusts the weight of each modality in feature fusion based on the uncertainty score Q′m, a normalized scalar in [0, 1] obtained by aggregating pixel-wise uncertainty scores Q∗m. The weight for each modality is defined by a weighting function that incorporates both uncertainty scores and inter-modality variance. The final fused feature is computed as a weighted sum of each modality's feature representation.

The paper also introduces two loss functions: The UQ loss Luncertainty penalizes the differences between feature representations in identified high-uncertainty regions... and prevents significant deviation from the representative features in the feature pool, and The modality consistency loss Lconsistency enforces feature alignment across modalities in high confidence (low-uncertainty) regions, ensuring coherent multi-modal fusion. The total loss is Ltotal = Lbaseline + αLuncertainty + βLconsistency.

Experiments were conducted on three public datasets: aiMotive, DeLiVER, and MFNet. The results show that CRUISE outperforms state-of-the-art (SOTA) methods by on average 4.87% in 3D object detection and 4.23% in semantic segmentation. On aiMotive, CRUISE delivers on average 3.82% better performance in all-point AP and 3.56% in 11-point interpolation AP compared to its baseline (aiMotive) without UQ. On DeLiVER, "CRUISE exhibits better performance compared to SOTA sensor fusion frameworks across all the scenarios... achieving on average 6.71% higher mIoU on the test set and 4.41% higher mIoU on the validation set over its baseline (CMNeXt) without UQ. On MFNet, CRUISE delivers on average 2.24% higher mIoU over the second-best result."

In the UQ methods comparison, "CRUISE outperforms the second-best model by on average 2.98% in all-point AP and 2.20% in 11-point interpolation AP on aiMotive, 1.56% on MFNet, and 4.24% higher mIoU on the testing set and 4.08% higher mIoU on the validation set of DeLiVER."

The ablation study shows that integrating the VLM leads to a substantial and consistent performance improvement, with an average increase of 3.42% compared to the variant without it. The prompt adapter further improves performance by an additional 0.81%. The fusion strategy ablation demonstrates that when all components are combined, our approach improves all-point and 11-point AP by 7.53% and 7.07%, respectively, under OOD conditions, and by 7.58% and 7.36%, respectively, under the ID conditions over the baseline.

The paper concludes: "We present CRUISE, a novel uncertainty-aware multi-modal sensor fusion framework that significantly improves the robustness and generalization capabilities of AV perception systems. Our primary innovation lies in the effective combination of two key components: VLM-guided uncertainty estimation and a dynamic adaptation mechanism. The VLM module leverages extensive prior knowledge to transform coarse reliability scores into fine-grained, context-aware uncertainty heatmaps. The dynamic adaptation mechanism is specifically designed to explicitly model and capture cross-modal dependencies, allowing the framework to intelligently balance contributions from various sensors and compensate for uncertainty across modalities. Through these integrated innovations, CRUISE successfully mitigates uncertainty propagation and achieves superior performance in complex, OOD scenarios."

Improvements to AI Systems:

1. Fine-Grained Uncertainty Estimation via Vision-Language Models (VLMs): Integrate a VLM-guided uncertainty module that generates pixel-level uncertainty heatmaps, replacing coarse modality-level scores. This enables dense prediction tasks (e.g., semantic segmentation) to identify and suppress unreliable regions with high spatial precision, improving robustness in out-of-distribution (OOD) scenarios.

2. Cross-Modal Dynamic Weighting: Implement a dynamic adaptation mechanism that computes per-modality weights based on aggregated uncertainty scores and inter-modality variance. This allows the system to automatically rebalance contributions from cameras, LiDAR, and radar in real time, compensating for sensor degradation (e.g., poor visibility) without manual tuning.

3. Uncertainty-Aware Feature Refinement: Use the generated uncertainty heatmap as a mask to suppress unreliable features before fusion. This prevents error propagation from noisy sensor data into downstream tasks, leading to more stable 3D object detection and segmentation in adverse conditions.

4. VLM-Guided Contextual Reasoning for Sensor Fusion: Adapt a pre-trained VLM (e.g., Anomaly GPT) via LoRA fine-tuning to interpret cross-modal features in a shared BEV space. The VLM’s global semantic understanding refines initial uncertainty estimates, capturing dependencies that local statistical methods miss—improving generalization to unseen environments.

5. Uncertainty and Consistency Losses for Training: Add two auxiliary losses: (a) an uncertainty loss that penalizes feature deviation in high-uncertainty regions, and (b) a modality consistency loss that enforces alignment in low-uncertainty regions. This jointly improves feature reliability and cross-modal coherence, accelerating convergence and boosting final accuracy.

6. Prompt-Adaptive Uncertainty Estimation: Incorporate a prompt adapter that dynamically generates structured prompts for the VLM based on the current input context. This allows the system to adapt its uncertainty reasoning to specific driving conditions (e.g., fog, night, rain) without retraining, enhancing flexibility across diverse datasets.

What the Improved AI System Can Do:

• Robust Perception in Adverse Conditions: Achieve significantly higher accuracy in 3D object detection (e.g., +4.87% over SOTA) and semantic segmentation (e.g., +4.23% over SOTA) under fog, rain, low light, and sensor noise, by intelligently down-weighting unreliable sensor data.

• Real-Time Adaptive Sensor Fusion: Dynamically adjust sensor contributions on-the-fly, ensuring seamless performance when one modality fails (e.g., camera glare or LiDAR occlusion), without requiring explicit failure detection.

• Generalize to Unseen Scenarios: Leverage VLM prior knowledge to handle OOD situations (e.g., novel object shapes or unexpected road layouts) that traditional fusion methods misclassify, improving safety in autonomous driving.

• Fine-Grained Reliability Mapping: Produce interpretable, pixel-level uncertainty heatmaps that highlight exactly where the system is uncertain, enabling downstream decision-making (e.g., path planning) to act cautiously in those regions.

• Efficient Training with Auxiliary Objectives: Reduce overfitting to training distributions by enforcing uncertainty-aware and cross-modal consistency constraints, leading to more stable validation performance and faster convergence.

Modern autonomous vehicles are equipped with multiple sensors, such as cameras, LiDAR, and radar, for comprehensive environmental perception. However, robust cross-modal feature fusion remains a critical challenge, as the reliability of each sensor varies significantly across diverse real-world driving conditions, including poor visibility and adverse weather. While uncertainty quantification (UQ) mitigates this issue by allowing models to prioritize reliable signals, existing uncertainty-aware fusion methods typically rely on simple feature-level uncertainty estimates and thus often fail to generalize effectively in complex, out-of-distribution scenarios. To address this limitation, we propose CRUISE, a novel uncertainty-aware cross-modal sensor fusion framework. CRUISE integrates a vision-language model (VLM)-guided UQ module that generates fine-grained, pixel-level uncertainty estimates. By leveraging the VLM's rich prior knowledge and superior contextual reasoning, our approach provides a highly informative guide for the fusion process. Furthermore, we introduce a dynamic adaptive mechanism that explicitly models and captures cross-modal dependencies, ensuring the framework fully exploits the inherent complementary nature of multi-sensor inputs.