Sign Language Recognition Using Original and Synthetic Depth Image Based Point Cloud Data Models

This paper investigates the use of point cloud data derived from both original and synthetic depth images for sign language recognition (SLR). The authors note that most SLR research relies on RGB images, which are susceptible to lighting and color variations, while depth images are less affected. However, depth datasets are limited. To address this, they use the Depth Anything V2 network to generate synthetic depth images from RGB images.

The study uses three public SLR datasets that contain both RGB and depth data: Real-time ASL Fingerspelling (Dataset-A, 24 gestures, 5 signers, 65,890 frames), KArSL (Dataset-B, 502 gestures, 3 signers, 75,300 videos), and AUTSL (Dataset-C, 226 gestures, 43 signers, 38,336 videos). Point clouds are created from both original and synthetic depth images using Open3D. Three data models are evaluated: frame-based point clouds, Point Gesture Maps (PGMs, which merge frames of a gesture), and LSTM-based models using features extracted from frame-based PointNet networks. PointNet architectures are used for classification, with modifications to the last layers for PGM and LSTM models.

For Dataset-A, which is static (no temporal dimension), only frame-based and PGM models are used. The best accuracy is achieved by the original depth frame-based PointNet at 94.16%, while the synthetic depth frame-based model reaches 73.40%. For PGM models, original depth achieves 70.59% and synthetic depth achieves 61.25%.

For Dataset-B, frame-based, PGM, and LSTM models are used. The original depth frame-based model achieves 90.57% accuracy, while the synthetic depth frame-based model achieves 59.90%. For PGM models, the original depth achieves 32.11% and the synthetic depth achieves 86.81%, showing a significant improvement for synthetic data. For LSTM models, original depth achieves 95.19% and synthetic depth achieves 97.56%, with synthetic data performing slightly better.

For Dataset-C, the original depth frame-based model achieves 44.28% accuracy, while the synthetic depth frame-based model achieves 29.23%. For PGM models, original depth achieves 14.09%, and the synthetic PGM model produced insignificant accuracy and was not included. For LSTM models, original depth achieves 68.43% and synthetic depth achieves 61.61%.

Overall, the results show that original depth-based point cloud models generally outperform synthetic ones, but synthetic depth data shows potential and in some cases (e.g., Dataset-B PGM and LSTM) performs better than original data. The authors conclude that synthetic depth data can be useful for SLR, and suggest future work on preprocessing to isolate hand/arm regions and testing additional datasets and synthetic depth generation methods.

Improvements to AI Systems:

1. Hybrid Depth-Augmented SLR Pipeline: Integrate a dual-stream architecture where one stream processes original depth point clouds and the other processes synthetic depth point clouds. Use a confidence-based fusion layer (e.g., learned attention) to combine features, leveraging the strengths of both (original for static accuracy, synthetic for temporal robustness). This improved system can achieve higher accuracy than either alone, especially in cross-domain settings (e.g., new signers or lighting changes).

2. Synthetic-to-Real Domain Adaptation Module: Add a domain-adversarial training step (e.g., gradient reversal) that aligns feature distributions between synthetic and original depth point clouds. This allows the AI to pre-train on large synthetic RGB-derived depth datasets (e.g., from YouTube videos) and fine-tune on limited real depth data. The improved system can generalize to unseen signers and environments with minimal real depth annotations.

3. Temporal Point Cloud Consistency Checker: Use the LSTM-based model’s superior performance on synthetic data (Dataset-B: 97.56%) to build a self-supervised consistency loss. The AI can compare predictions from original and synthetic depth streams over time, penalizing disagreements. This improves temporal coherence and reduces flicker in sign recognition, enabling more stable real-time gesture interpretation.

4. Adaptive Depth Quality Estimator: Train a small regression head on the PointNet features to predict per-frame depth reliability (e.g., based on noise or occlusion). The AI then dynamically weights original vs. synthetic depth frames during inference. This system can automatically switch to synthetic depth when original depth is noisy (e.g., reflective surfaces, distant hands), improving robustness in uncontrolled environments.

5. Synthetic Data Generation for Rare Gestures: Leverage the finding that synthetic depth helps in large-vocabulary datasets (Dataset-B) by using Depth Anything V2 to generate synthetic depth for any RGB-only sign language video (e.g., from online dictionaries). This expands training data for rare or unseen gestures, allowing the AI to recognize a broader vocabulary without additional depth sensors.

6. PGM-Specific Synthetic Enhancement: Since synthetic PGMs significantly outperform original PGMs on Dataset-B (86.81% vs. 32.11%), design a model that explicitly generates synthetic depth PGMs from RGB PGM inputs. This improved system can perform isolated sign recognition using only RGB cameras, making it deployable on standard smartphones or webcams, while retaining near-state-of-the-art accuracy.

7. Cross-Dataset Calibration Network: Train a lightweight calibration network that maps synthetic depth features to original depth feature space (or vice versa) using a small paired dataset. This allows the AI to combine multiple SLR datasets (e.g., train on AUTSL synthetic, test on KArSL original) without performance collapse, improving cross-dataset generalization.