A deep learning-based framework for correcting erroneous character-level Bengali sign images

Abstract

People who cannot hear or speak rely on sign language as the primary source of communication. Sign language is generally expressed at a fast pace, therefore errors and miscommunication may happen frequently. Our thesis introduces a deep learning–based framework for correcting erroneous character-level Bengali sign language images. The main focus of our research is correctly mapping incorrect bengali character level sign gestures to their closest semantically accurate signs. A dataset consisting of 36 classes for both correct and potential incorrect hand signs were generated for bengali characters. The proposed framework utilizes convolutional neural networks(CNNs) along with triplet loss to extract discriminative embeddings. These embeddings are later represented on a vector space where metric distance from incorrect images are used to map them to the correct gestures. Our primary goal is to develop a system where the intended meaning is correctly delivered even in case of improper hand gestures. Overall, 9 models were used to assess the proficiency of the idea for correcting erroneous bengali sign characters. ResNet50 delivered the most excellent results with an accuracy of 97.6% on a 200 epoch experiment. Further ablation studies also resulted in other models performing well. VGG16 had an accuracy of 94.4% and EfficientNetB0 delivered an accuracy score of 96.4%. The research concluded that longer epochs provided significant increases in accuracy for both black and normal backgrounds. Both KNN and centroid based distance were used as similarity based mapping approaches in order to compare the accuracy of the models under various changes in hyper-parameters. The centroid performed better throughout all the experiments, Thus, the overall results highlight the successful mapping of incorrect sign gestures to their appropriate correct classes.