Improving Bangla sign language detection from thermal imagery: leveraging thermal heatmap-induced transfer learning and deep neural networks

Abstract

This research explores fusion of deep learning for sign language recognition and thermal imaging, focused on advancing communication systems for the hearing impaired. Recognition of sign language is a crucial technology for improving accessibility and the use of thermal images light up new possibilities for gesture or sign recognition in different lighting conditions.To address the challenges associated with recognizing Bangla sign language, we have collected and built a novel thermal image dataset using the ABF Astron Infrared Camera F3.20. Additionally, we created 49 distinct classes for Bangla sign gestures using a colormap filtered version of the dataset to enhance feature visibility. The study employed transfer learning to retrain pre existing neural networks on both the colormap and thermal datasets. Three prominent deep learning models, ResNet, DenseNet, and Inception, were selected for this task due to their proven effectiveness in image classification tasks.These models were first trained on the colormap dataset. Subsequently, we tested the models on the original thermal dataset, using transfer learning to refine the learned features. This process showed the potential for improved gesture recognition when utilizing both thermal and color mapped images.The results highlight the importance of combining transfer learning with advanced neural networks to enhance recognition systems. With accuracies of 95%, 90%, and 98% across different models, the findings demonstrate the promising application of thermal imaging in improving the reliability and accessibility of sign language recognition technologies. This approach could offer more sturdy solutions for real-time sign language translation, particularly in challenging environmental conditions.