Real-time Bangladeshi Sign Language recognition and natural language interpretation using hybrid deep learning and LLMs

Abstract

Deaf and hard-of-hearing communities in Bangladesh face a persistent communication gap: existing sign language recognition systems process hand gestures but ignore the facial expressions and head movements that carry grammatical meaning in Bangladeshi Sign Language (BdSL). This study introduces BdSL-NMM, the first BdSL dataset annotated with non-manual marker labels alongside word-level glosses, covering 62 sign classes and 5 expression categories across 4,170 recordings from seven native signers. To evaluate on this dataset, a multi-stream Transformer architecture called SignNet-V2 was developed, which simultaneously recognises sign words and non-manual expression markers from skeletal landmark sequences. The system processes four input modalities, namely body pose, left hand, right hand, and face mesh, through dedicated stream-specific encoders, cross-stream attention fusion, hierarchical temporal encoding, and a multi-task classification head. All models were trained and evaluated under leave-one-signer-out cross-validation, where the test signer never appeared during training, providing a realistic measure of cross-signer generalisation. Under this protocol, SignNet-V2 achieves 38.44% top-1 word recognition accuracy. For expression recognition, a 56-dimensional framework of normalised geometric and temporal features achieves 66.39% overall accuracy under signer-independent evaluation, with neutral and negation recall of 97.22% and 91.18% on the unseen test signer. A second interpretation stage passes recognised signs and expression tags to Google Gemini, which generates grammatically correct Bengali sentences. This work establishes the first published benchmarks for simultaneous word and non-manual marker recognition in BdSL under signer-independent evaluation.