Computer vision-based Bengali sign language to text generation

Abstract

In the whole world, around 7% of people have hearing and speech impairment problems. They use sign language as their communication method. People from various countries use a variety of sign languages. As an example, there are ASL, CSL, JSL, etc. Even in our country, there are lots of people born with hearing and speech impairment problems. So, our primary focus is to work for those people by converting Bangla sign language into text. There are already various projects on Bangla sign language done by other people. However, they focused more on the separate alphabets and numerical numbers. That is why we want to concentrate on Bangla word signs since these people prefer to communicate using words or phrases rather than alphabets. There is not any proper database for Bangla word sign language, so we are making a database for Bangla word sign language for our work. In recognition of sign language (SLR), there usually are two types of scenarios: isolated SLR, which takes words by word and completes recognize action, and the other one is continuous SLR, which completes action by translating the whole sentence at once. We are working on isolated SLR. We introduce a method where we are going to use PyTorch and YOLOv5 for a video classification model to convert Bangla sign language into the text from the video where each video has only one sign language word. Here,we have achieved an accuracy rate of 76.29% on the training dataset and 51.44% on the testing dataset. We are working to build a system that will make it easier for hearing and speech-disabled people to interact with the general public.