WILD SAM: A Simulated-and-Real Data Augmentation for Autonomous Driving Perception under Challenging Weather

The gist The proposed WILD SAM framework, which integrates physics-based simulation and pseudo-label denoising within an Unsupervised Domain Adaptation (UDA) framework, effectively mitigates domain shift for LiDAR-based 3D object detectors under adverse weather conditions.

Introduction and Problem Statement

LiDAR-based 3D object detection becomes challenging under adverse weather conditions such as rain, snow, and dense fog because these conditions degrade and attenuate LiDAR returns causing detectors to generate unreliable predictions Such inaccuracies raise serious safety risks especially for vulnerable road users like pedestrians and cyclists Existing approaches typically rely on data augmentation or feature-level domain adaptation for 2D object detection However, these methods have not been exploited by weather-based domain adaptation approaches due to the noisy nature of such labels generated under harsh weather conditions

Proposed Solutions

The paper proposes two new approaches to mitigate this weather-induced domain shift First, they propose a Weather-Induced pseudo-Label Denoising (WILD) framework that filters noisy pseudo labels generated by real data captured under adverse weather conditions Second, they develop a novel hybrid training methodology, WILD SAM, that exploits both pseudo-label denoising and simulation-based training solutions while using real-data from the target harsh-weather domain

WILD SAM Training Methodology

Under WILD SAM-based training, the framework follows an unsupervised domain adaptation (UDA) strategy where an initial 3D LiDAR-based detector is trained on labeled source data The process involves three main steps:

1. Training Baseline Model: Train an initial 3D object detector on labeled source-domain frames and their ground-truth annotations

2. Auxiliary Data Generation: This step generates two types of auxiliary datasets:

**: Weather-Induced Pseudo-Label Denoising (DALI) Augmented frames, which are generated by running the pre-trained 3D detector on unlabeled target domain frames and applying pseudo-label denoising to generate Weather-Induced pseudo labels. The paper adapts the rayconstrained pseudo point-cloud generation (RC-PPCG) method from DALI [16] to handle harsh weather target domain data. This involves projecting points within target-domain bounding boxes onto a dense reference model via ray tracing. The paper constructs a target-domain database from the denoised pseudolabels by rearranging the points within each pseudo-label. This target-domain database is then used in WILD frames training separately from the ground-truth database of the source domain. The paper notes that even if a false positive pseudo-label mistakenly corresponds to a non-object region, the associated points are rearranged to follow the geometry distribution of the intended target object class. 2) Physics-based Simulated Augmentation and Modeling (SAM) frames, which use LISA [20] to synthesize harsh-weather conditions like snow and rain. For each labeled source frame, a corresponding simulated version is generated using LISA, where the weather intensity is controlled by the precipitation-rate parameter τ (mm/hr). The original ground-truth annotations remain valid and are kept aligned with the simulated frames 3. WILD SAM-based Training with Source & Auxiliary Data: The final detector is retrained using three independent sets of lidar point-cloud frames: (a) source-domain frames with their ground-truth labels, (b) augmented target domain frames with denoised pseudo labels using WILD, and (c) Physics-based Simulated Augmentation and Modeling (SAM) frames with source-domain ground-truth labels. Each dataset is further augmented independently with foreground object sampling augmentations such as random flips or random rotations. This setup triples the training batch size under the full WILD SAM framework while keeping the model architecture and test-time inference cost unchanged. The paper summarizes that WILD exploits real data and enables training with less noisy point cloud frames containing more reliable pseudo labels. 3) Results and Discussion: Experiments on the MSU-4S dataset show that both WILD and WILD SAM consistently improve robustness when tested on real data captured under adverse weather Table II demonstrates that WILD SAM provides significant gains for vulnerable road users such as pedestrians and cyclists. The results show that both WILD and WILD SAM consistently improve detection accuracy under harsh weather conditions by more than 13%. Table IV reports results for both PV-RCNN++ and Voxel-RCNN, showing that most of the detection gain comes from the WILD augmented frames Conversely, Table IV(a) shows that using synthetic data alone can sometimes slightly hurt performance whereas combining synthetic samples with WILD augmented frames can lead to the highest overall increase in mAP. The paper concludes that precise point–box matching plays a critical role in achieving robust detection accuracy improvement for different weather conditions. 4) Conclusion: In this study, the authors demonstrated the domain shift caused by adverse weather conditions for two widely used LiDARbased 3D detectors, PV-RCNN++ and Voxel-RCNN To address this domain gap, they propose WILD SAM, a hybrid approach that integrates physics-based simulation with pseudolabel denoising within an Unsupervised Domain Adaptation (UDA) framework to enhance the robustness of LiDAR-based detectors Their experiments on the MSU-4S dataset validate that WILD SAM-based training effectively mitigates domain shift while maintaining the same computational complexity during inference. The paper also suggests that snowy and rainy conditions share similar characteristics, enabling WILD SAM-trained models to generalize effectively across different types of harsh weather, such as rain and snow. The final conclusion is that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish. The paper concludes that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish. The paper concludes that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish. The paper concludes that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish. The paper concludes that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish. The paper concludes that the combination of synthetic data with WILD augmented frames can lead to the highest overall increase in mAP. The paper also notes that increasing τ generally improves mAP up to a certain level after which detection accuracy gains begin to diminish.

1. The WILD SAM framework can filter noisy pseudo-labels by employing rayconstrained pseudo point-cloud generation (RC-PPCG) method to denoise pseudo-labels by projecting points within target-domain bounding boxes onto a dense reference model. This ensures that the training data used for target domain adaptation is more reliable, mitigating the issue where pseudo labels are noisy and contain a significant number of false positives.

2. The WILD SAM methodology creates a highly diverse training set by combining three independent data banks: "source frames with their ground-truth labels, (b) augmented target domain frames with denoised pseudo labels using Weather-Induced pseudo-Label Denoising (WILD), and (c) Physics-based Simulated Augmentation and Modeling (SAM) frames with source-domain ground-truth labels. This triple augmentation strategy maximizes input diversity for the final 3D object detector," leading to improved robustness.

3. The resulting WILD SAM system can achieve a performance improvement of Average Precision (AP) up to 13% over baselines when tested on the MSU-4S dataset across rainy and snowy scenarios, specifically showing gains for both LiDAR detectors like Voxel-RCNN and PV-RCNN++.