BDSLI: a hybrid CNN-transformer model for bengali sign language interpretation

Abstract

Sign Language Recognition (SLR) is a widely explored field of study all over the world, yet progress remains limited for certain languages, including Bengali. In this study, a hybrid model (CNN + Transformers) is proposed to recognize isolated Bengali sign words and generate meaningful sentences. To the best of our knowledge, this specific combination has not been explored before. As a prerequisite for training the model, a video dataset is constructed comprising 62 Bengali sign words, with 250 videos per class. In addition, a separate test dataset is created to evaluate the model performance on unseen data. Additional hybrid models, including CNN+LSTM, CNN+BiLSTM, and CNN+GRU, are trained and evaluated alongside the proposed approach. Furthermore, widely recognized architectures such as LSTM, GRU, TCN, and Transformers are also implemented to demonstrate the superiority of the proposed model for the training dataset. In this study, it will be demonstrated that the chosen model can achieve an impressive 99.58% accuracy (with 99.48% validation accuracy) for the training dataset and 98.65% accuracy for the test dataset. These results surpass those of all other models that are trained and evaluated in this study. Following a comprehensive evaluation, this research intends to deploy the trained model in a web application to illustrate its effective performance.