Toward Deployable Bangla Sign Language Recognition with Expert-Validated Data and a Lightweight Attention-Based Model

The paper addresses automatic recognition of Bangla Sign Language (BdSL), the primary means of communication for the deaf and hard-of-hearing (DHH) community in Bangladesh. The authors note that approximately 430 million people worldwide, over 5% of the global population... live with disabling hearing loss and that in Bangladesh the prevalence of hearing impairment is nearly 9.6%, almost double the global average, and an estimated 2.5 to 3 million people rely on Bangla Sign Language (BdSL) daily. BdSL comprises 38 hand signs representing the 51 letters of the Bangla alphabet, in both one-handed and two-handed forms.

The authors identify three key gaps in existing BdSL research: (1) expert-verified data is scarce — existing public datasets (BDSL49, BdSL36, KU-BdSL, BAUST Lipi, BdSL47, Shongket) share few contributors (typically 10 to 42), predominantly non-signing volunteers, no complete expert linguistic verification... and controlled capture that does not reflect real-world diversity; (2) prior models are either heavyweight pretrained architectures not optimized for efficient deployment, or custom models evaluated only on their own data without cross-dataset or ablation validation; and (3) evaluation protocols are permissive — "results are almost universally reported on class-stratified splits in which images of the same signer appear in both partitions, so the reported figure silently includes what the model has learned about individual people."

The paper's four contributions are: the RSBdSL38 expert-validated dataset; a lightweight attention-based CNN of 298,470 parameters; systematic validation via stage-wise depth ablations, component controls, and explainability analysis; and deployment-facing evaluation including signer-independent protocols, zero-shot cross-corpus transfer, and on-device efficiency measurements. The work investigates five research questions (RQ1–RQ5) covering accuracy parity with pretrained models, cross-dataset generalization, ablation of components, dataset ecological validity, and whether predictions follow the signing hand.

RSBdSL38 (Real-Signer Bangla Sign Language, 38 classes) was designed around three principles: authentic signers, expert verification, and environmental diversity. Images were collected from real signers at three special-needs schools in three different districts of Bangladesh, where BdSL is the daily medium of communication, supplemented by additional volunteers. In total, 36 participants contributed (17 male, 19 female) in two age bands: 21 aged 8–15 years, the deaf and hard-of-hearing pupils recruited at the schools, and 15 aged 20–28 years, the adult volunteers. All signs were captured with smartphone cameras "framed on the hand region and articulated naturally, without constraining hand orientation or requiring uniform backgrounds, so the corpus contains the classroom walls, furniture, floor tiling, and variable illumination of the recording sites."

The dataset covers "the complete canonical BdSL manual alphabet: 38 sign classes jointly representing the 51 letters of the written Bangla alphabet... Of the 38 signs, 9 are vowels (shoroborno) and the remaining 29 consonants and diacritic modifiers (banjonborno), with written letters that share an identical manual form grouped under a single sign. After a manual quality-screening pass to remove blurred or poorly framed images, the complete dataset was independently reviewed by an experienced senior Bangla Sign Language teacher at one of the participating schools... Images identified as mislabelled or incorrectly articulated were flagged, and the corresponding corrections were applied by the research team. The validated corpus contains 10,874 images, distributed approximately uniformly across the 38 classes (≈286 per class) with mean 286.2 per class, standard deviation 15.5, and a maximum-to-minimum imbalance ratio of only 1.25. The data is partitioned by class-stratified sampling into training, validation, and test sets of 8,794 / 977 / 1,103 images (80.9% / 9.0% / 10.1%)."

The authors argue that RSBdSL38 is unique in combining full-alphabet coverage, genuine BdSL signers from multiple geographic sites, uncontrolled capture conditions, and complete expert validation. They also note that Because participant identity is recorded for every image, it additionally supports the signer-independent protocol of Section 6.3, which none of the compared datasets documents.

The proposed network maps a 224 × 224 × 3 RGB image to a probability distribution over the K = 38 classes using only 298,470 parameters (292,262 trainable and 6,208 non-trainable batch-normalization statistics, occupying 1.14 MB in single precision). Four design principles govern: bottleneck factorization with grouped convolutions; convolutional block attention to direct limited capacity towards the signing hand rather than the background; a dedicated multi-scale depthwise hand-feature block capturing finger-level and palm-level structure simultaneously; and smooth activations with strong regularization.

The architecture consists of:

• Stem: a strided 5 × 5 convolution with 32 filters followed by batch normalization, Swish, and overlapping max pooling reducing resolution by a factor of four to 56 × 56 × 32.

• Grouped Bottleneck Residual Attention Blocks: each of four stages contains two residual blocks compressing to C/2 channels, processing with grouped 3 × 3 convolutions with group counts gi ∈ 2, 4, 8, 16, and re-expanding. The group count scales with the stage width... keeping the per-group channel count close to constant across depth.

• Channel attention (CBAM-style): Global average and global max pooling produce two channel descriptors that are passed through a shared two-layer bottleneck multi-layer perceptron (MLP) with reduction ratio r = 16, summed, and squashed, costing only 2C2/r weights.

• Spatial attention: A single-channel spatial saliency map is then computed with a bias-free 7 × 7 convolution and sigmoid gating, costing 49C weights.

• Hand-Feature Block: a 1 × 1 projection is followed by two parallel depthwise branches with 3 × 3 and 5 × 5 kernels, whose outputs are concatenated, fused, and regularized with stage-wise increasing spatial dropout rates ρi ∈ 0.05, 0.10, 0.15, 0.20.

• Dual-Pooling Classification Head: concatenates global max pooling and global average pooling into a 256-dimensional vector, processed by two fully connected layers with Swish, batch normalization, and dropout (p = 0.3) to a 38-way softmax.

The four stages operate at 562 × 32, 282 × 64, 142 × 96, and 142 × 128 resolutions, and the resolution is deliberately held at 14 × 14 through the last two stages: further downsampling would collapse the finger detail on which several classes depend.

Training uses stochastic gradient descent (SGD) with Nesterov momentum (µ = 0.9), initial learning rate η0 = 10−2 with reduce-on-plateau halving, early stopping, batch size 32, l2 regularization (λ = 10−5), and He normal initialization. Data augmentation includes a random rotation of up to ±15°, a random zoom of ±10%, and a random translation of ±10% along each axis plus mild photometric jitter. Horizontal and vertical flipping are deliberately excluded: BdSL signs are articulated with respect to a dominant hand and a fixed palm orientation, so a mirrored image is not a valid example of the same class.

The proposed model achieves 96.37% test accuracy on the 1,103-image RSBdSL38 test set, with weighted precision 96.50%, recall 96.37%, and F1-score 96.38%. Retraining over five seeds gives a mean of 95.72% ± 0.54% accuracy with the full range spans only 1.36 points (95.10 to 96.46%). Per-class F1 ranges from 0.87 to 1.00, with four classes (11, 17, 21, 32) classified perfectly and 28 of 38 at or above 0.95. Only 40 of 1,103 test images are misclassified, and the largest off-diagonal entry is three images (class 12 predicted as class 36).

Signer-independent evaluation: Holding out 6 of the 36 participants... entirely from training, the model attains 85.18% (weighted precision 86.42%, F1 85.23%) over the 1,417 held-out test images, an 11.19 percentage-point drop from the stratified 96.37%. The authors state: This is the measured cost of signer leakage: roughly a ninth of the stratified accuracy is attributable to recognizing familiar signers rather than the signs themselves. The degradation analysis shows the additional errors arise from signer-specific variation in articulation rather than intrinsic ambiguity between sign pairs.

Comparison with pretrained baselines: Nine architectures spanning five efficient design families (MobileNetV4, MobileViT/MobileViTv2, EfficientNetV2, EfficientFormerV2, GhostNetV2) were fine-tuned under an identical protocol. The accuracy band across all ten models is narrow (95.33 to 97.45%) — MobileNetV4-Hybrid-M leads at 97.45%, while the proposed from-scratch model reaches 96.37%: above GhostNetV2-1.0 (95.33%) and MobileViTv2-1.0 (96.18%), within 0.11 percentage points of EfficientNetV2-B0 and 0.17 of MobileViT-S, and within 1.08 of the best model, MobileNetV4-Hybrid-M, at 33× fewer parameters. Against SA-CNN, the only lightweight custom BdSL architecture in the literature (671,942 parameters, 93.47%), it is 2.25× smaller and 2.90 percentage points more accurate.

The identical architecture, with no structural change and the same training protocol, was therefore retrained from scratch on six public BdSL benchmarks: KU-BdSL (98.33%), BdSL47 (97.81%), Shongket (95.15%), BdSL-38 (94.57%), BAUST Lipi (93.56%), and BdSL36 (92.95%). On a merged corpus combining RSBdSL38 with BdSL-38, BdSL47, and KU-BdSL, it achieves 97.04% accuracy. On BdSL-38 it attains 94.57% as a single 0.30M-parameter network, against 96.62% from a five-model ensemble of heavyweight pretrained CNNs.

For zero-shot transfer, the RSBdSL38-trained reference model was evaluated directly on the full public distribution of BdSL-38... (12,581 images, identical 38-class label scheme, verified one-to-one) without any fine-tuning, attaining 76.25% zero-shot accuracy (macro F1 76.00%, weighted F1 76.03%). The 18.32-point gap versus the retrained 94.57% isolates the domain shift. The authors note two mutually confused classes (15 and 23) as a genuine visual near-degeneracy in how BdSL-38 renders the two articulations rather than an annotation mismatch.

At 132.7 M MACs, the proposed model is the cheapest in the comparison, needing 1.3× fewer MACs than the next-lightest baseline (GhostNetV2-1.0) and 21.7× fewer than the heaviest (EfficientNetV2-S). Parameter ratios range 8.5–68× fewer than baselines. Post-training quantization compresses the model to 0.43 MB (dynamic-range TFLite) and 0.46–0.48 MB (full-integer INT8), the latter actually improving test accuracy fractionally to 96.46%. On a commodity Android smartphone (Snapdragon 7+ Gen 3, 12 GB RAM), the mean latency is 3.98 ± 0.09 ms (median 3.96, 95th percentile 4.16), approximately 251 frames per second, with peak memory 15.5 MB, initialization 13.8 ms, and 241 of 315 operators hardware-delegated.

Stage-wise depth ablation (Study A): All 14 non-empty stage-removal configurations were evaluated against a full-model baseline (D0, 96.55%). "Every single-stage removal produces a material loss... Stage 2 is the least damaging omission: D2 reaches 88.94%, 7.61 percentage points below D0... Stage 4 is the most important: D4 falls to 75.79%, a 20.76 percentage-point deficit. Pairwise removals reveal that which stages remain matters more than raw model size: D9 retains only 123,686 parameters yet reaches 83.05%, while D5 has 241,030 parameters but only 79.87%. Removing all stages leaves just 7.25% accuracy. The authors conclude no stage-removal variant remains close to D0," so quantization rather than architectural trimming is used for deployment.

Component controls (Study B): Three parameter-matched controls against the same 298,470-parameter baseline: removing augmentation drops accuracy from 96.55% to 93.38% (−3.17 percentage points); removing dropout lowers accuracy to 94.92% (−1.63); replacing Swish with ReLU gives 95.56% (−0.99). Because the parameter count is identical, none of these gaps reflects model size. The consolidated comparison shows the component controls span 0.99 to 3.17 percentage points, the 14 stage removals 7.61 to 89.30, so preserving the full stage hierarchy matters far more.

Grad-CAM and Grad-CAM++ maps applied to the Stage-4 hand-feature block show the attribution peak falls on the signing hand in all eight cases, specifically on the configured fingers rather than the wrist or forearm — including for samples "photographed against strongly structured backgrounds: saturated red, green, and pink lockers... a whiteboard covered in handwritten Bangla characters... a tiled floor with regular grout lines... and a wall crossed by a magenta stripe. Stage-wise maps show Background rejection is therefore established between Stages 2 and 3."

Failure analysis of five misclassified images shows both maps fall on the same hand region in every case, indicating a discrimination failure rather than a localization failure. The confidence distribution over the 40 errors reveals mean probability 0.81 on the wrong class, 19 of the 40 above 0.9 and 11 above 0.99.

Faithfulness was quantified with deletion and insertion measures over 200 test images: "deleting pixels in Grad-CAM order collapses the predicted-class probability roughly 2.6× faster than random deletion (deletion AUC 0.081 versus 0.211), and restoring them recovers the prediction far sooner (insertion AUC 0.888 versus 0.605). Grad-CAM is reported as the more faithful of the two on every measure — Grad-CAM++'s sharper maps are the less faithful, because restricting the input to the Grad-CAM++ region discards contextual evidence the network in fact uses. A cascading weight-randomization sanity check confirms the maps collapse to a near-uniform field with no correspondence to the hand, demonstrating localization is a property of the learned weights rather than of the input's edge content."

The paper concludes: "We introduced RSBdSL38, an expert-validated Bangla Sign Language dataset of 10,874 images collected from real signers across three regions of Bangladesh, together with a lightweight attention-based convolutional network of only 298,470 parameters (1.14 MB, 132.7 M multiply–accumulate operations)... it quantizes to a 0.48 MB INT8 model running at 3.98 ms per image on a commodity smartphone, generalizes across six public BdSL benchmarks (92.95 to 98.33%) and a merged four-dataset corpus (97.04%) without any architectural change, and transfers zero-shot to BdSL-38 at 76.25%."

Acknowledged limitations include: RSBdSL38 covers static alphabet signs only; dynamic word- and sentence-level signing needs temporal modeling outside this architecture's scope; energy consumption and sustained-load thermal behaviour are not measured; the baselines are not re-evaluated under the signer-independent protocol; the component study covers the training recipe and the activation function but not the individual attention modules; and the explainability analysis is limited by the 14 × 14 attribution resolution and is not complemented by a perturbation-based method such as SHAP. Future work will address these, extend RSBdSL38 to dynamic word-level signing, and characterize the energy and sustained-load thermal behaviour of the quantized model on target hardware. The data…

Improvements to AI systems:

• Architecture design: Use a lightweight grouped bottleneck residual network with channel and spatial attention, a multi-scale depthwise hand-feature block, and dual-pooling classification head. This achieves 96.37% accuracy with only 298,470 parameters, 132.7 M MACs, and 1.14 MB size — enabling efficient on-device deployment.

• Regularization and training recipe: Apply staged spatial dropout, smooth Swish activations, batch normalization, SGD with Nesterov momentum, and augmentation excluding horizontal/vertical flips (because sign classes are mirror-sensitive). This improves accuracy by up to 3.17 points over parameter-matched controls.

• Signer-independent evaluation: Train with a participant hold-out protocol so the model is measured on unseen signers, not memorized individuals. This reveals an 11.19-point drop from stratified accuracy and forces the model to learn sign identity rather than person appearance.

• Cross-dataset generalization: Retrain the same architecture on multiple public sign-language benchmarks and evaluate zero-shot transfer across corpora to detect domain shift, annotation inconsistencies, and visual near-degeneracies between classes.

• Post-training quantization: Use dynamic-range and full-integer INT8 quantization to compress to 0.43–0.48 MB while slightly improving test accuracy (96.46%).

• Explainability-driven validation: Use Grad-CAM / Grad-CAM++ with deletion/insertion AUC and cascading weight-randomization sanity checks to verify that the model focuses on the signing hand, not backgrounds, and to distinguish localization failures from discrimination failures.

• Ablation-based architecture verification: Perform exhaustive stage-removal and component-control ablations to determine which depth stages and modules are most critical, avoiding unnecessary architectural complexity before deployment.

What the improved AI system can do:

• Recognize Bangla Sign Language alphabet signs at 96.37% accuracy in real-world, uncontrolled environments (classroom walls, furniture, variable lighting, structured backgrounds).

• Run on commodity Android smartphones at approximately 251 frames per second (3.98 ms per image) with 15.5 MB peak memory and 241/315 hardware-delegated operators.

• Transfer to six different Bangla sign-language datasets with 92.95–98.33% accuracy without any architectural change, and perform zero-shot recognition on an unseen corpus at 76.25% accuracy.

• Maintain high accuracy after INT8 quantization (96.46%), making it suitable for energy-constrained mobile and edge devices.

• Localize the signing hand and configured fingers even under strongly cluttered backgrounds, with quantified faithfulness (deletion AUC 0.081 vs. 0.211 random; insertion AUC 0.888 vs. 0.605).

• Avoid signer leakage: perform credible signer-independent recognition at 85.18% on completely unseen participants, rather than inflated stratified metrics.

• Identify dataset weaknesses, ambiguous sign pairs, and annotation problems automatically via cross-corpus confusion analysis, enabling future dataset correction and expansion.

Deaf and hard-of-hearing people in Bangladesh communicate mainly through Bangla Sign Language (BdSL). Automatic BdSL recognition on personal devices could widen access to education and services. Existing systems use controlled-setting datasets without expert verification and heavyweight pretrained backbones unsuited to on-device use. We introduce RSBdSL38, 10,874 expert-validated images spanning all 38 BdSL hand signs, representing the 51 letters of the Bangla alphabet, recorded from real signers at three special-needs schools across Bangladesh. We propose a lightweight attention based convolutional network of 298,470 parameters, built from grouped bottleneck residual blocks, channel and spatial attention, a multi-scale depthwise hand-feature block, dual pooling, and Swish activations. Trained from scratch, it attains 96.37% accuracy (95.72% +- 0.54% over five seeds), within 1.08 percentage points of the best of nine ImageNet-pretrained efficient architectures under an identical protocol, using 8.5 to 68x fewer parameters and 1.3 to 21.7x fewer MACs. Retrained, it reaches 92.95 to 98.33% on six public BdSL benchmarks, 97.04% on a merged corpus, and 76.25% zero-shot on BdSL-38. Removing any architectural stage costs 7.61 to 89.30 points, against at most 3.17 for the training recipe. Grad-CAM with deletion-insertion and weight-randomization checks confirms that predictions follow the signing hand. A signer-independent split holding out 6 of 36 signers yields 85.18%. Quantized to 0.48 MB, it runs at 3.98 ms per image within a 15.5 MB footprint on a commodity smartphone. Together, RSBdSL38 and our from-scratch model turn benchmark accuracy into deployable accessibility at a fraction of pretrained-backbone cost; dataset, code, and models are released.