Enhancing bidirectional sign language communication: integrating YOLOv8 and NLP for real-time gesture recognition

Abstract

Sign language is a visual language expressed through physical movements instead of spoken words. Hands, eyes, facial emotions, and movement are all used as visual clues in this language. Although many hearing people also use sign language, it is predominantly utilized by those who are deaf or hard of hearing. The grammar and structural norms of sign language have developed over time, much like those of any spoken language. Most often we see that normal people who don’t know sign language, struggle while communicating with people having hearing issues. It creates problems for the people to communicate and share their issues in case of emergency. The aim of this research is to take American Sign Language (ASL) data through real time camera footage and be able to convert the data and information into text. Moreover,the model can also convert text into sign language and show it to people who understand sign language, which can help us break the language barrier for the people who are in need. For recognising American Sign Language (ASL), we will be using the You Only Look Once(YOLO) model and Convolutional Neural Network (CNN) model. YOLO model is run in real time and automatically extracts discriminative spatial-temporal characteristics from the raw video stream without the need for any prior knowledge, eliminating design flaws.For converting text input to sign language, we will make a framework that will take a sentence as input, identify keywords from that sentence and then show a video where sign language is performed with respect to the sentence given as input. This framework will also function in real time.