Sign language recognition using CNN and OpenPose and making 3D model of sign language

Abstract

According to the World Health Organization (WHO), 466 million people over the world have impairing hearing misfortune (over 5 percent of the total populace), of whom 34 million are kids. There is an immense correspondence gap between ordinary individuals and hard of hearing and quiet people. Some guardians are unconscious of sending their hearing and talking hindered kid to class or some of them feel bashful. So for learning gesture based communication is hard for those youngsters and some incapacitated grown-up individuals are additionally don't know gesture based communication appropriately. This circumstance causes us to think of our thought. Our venture is making 3D model for all alphabets and numbers as well as some mostly used words and sentences of Bengali language and train the individuals as a virtual teacher. The model will help the individuals until he will do 100 percent correctly. Here for sign language recognition we use algorithm called Inception v3 which is an extended version of CNN (Convolutional Neural Network) and for activity recognition we use OpenPose. Tensor ow is used for coding as this is vastly used for machine learning and deep learning. We use video as our dataset and create 3d avater to teach user sign language. The 3d avatar is created using OpenPose unity plugin. Most of the researches based on sign language are hand gesture based. Unfortunately, sign language not only consist of hand gesture, It also includes face gesture, eye's gesture. And we are taking all of these things in consideration and we are trying to implement all of these. While detecting our Bangla numbers and letters in sign language we have got 89 percent accuracy.