A deep learning approach of translating speech into 3D hand sign language (ASL)

Abstract

Automatic Sign Language Recognition (ASLR) is a rapidly advancing field that leverages computer vision and machine learning to interpret sign language gestures and convert them into text or spoken language. The goal is to facilitate communication between deaf and hearing individuals and to enhance accessibility in various domains. Each country typically has its sign language, such as American Sign Language (ASL) in the United States and British Sign Language (BSL) in the UK, with International Sign (IS) used in global contexts. While substantial research has focused on translating sign language to text, the reverse process of translating text to sign language still needs to be explored. This discrepancy highlights an essential gap in accessibility technologies for the deaf and hard-of-hearing communities. Text-to-sign language translation involves converting written or spoken language into corresponding sign language gestures, which is inherently complex due to sign language’s rich, visual, and spatial nature. Our research aims to develop a novel system for converting spoken language into 3D hand sign language, bridging the communication gap between the hearing and deaf communities. This innovative approach involves two main components: automatic speech recognition (ASR) and 3D hand sign generation.