Advancements in real-time sign language translation

Abstract

The need for effective sign language recognition and translation has become more critical to create a more inclusive society that addresses the communication concerns of the Deaf community. In recent years, the field has seen a revolutionary progress arc, spearheaded by the development of transformative Deep Learning based approaches such as reinforcement learning, spatio-temporal residual networks, temporal convolution modules, iterative alignment networks, and attention mechanisms. Yet, vision-based real time continuous sign language recognition (CSLR) continues to face several application challenges, encompassing its visual, sequential, and alignment modules. As such, we propose an end-to-end training model inspired by the recent successes of transfer learning and attention-based mechanisms in particular to achieve new state-of-the-art performance on current benchmarks. Our paper includes two variations of approaches to dealing with continuous sign language videos: a classification approach and a translation generation approach. It eventually highlights the suitability of the translation-based approach for this domain of research. A comparative analysis between Classification based and generation based model highlights the superior efficiency and accuracy of the latter, making it the most suitable model for real-time, sentence-level sign language-to-text translation. Furthermore, our optimized inference strategy significantly reduces latency, ensuring real-time translation speeds, which is a crucial requirement for practical applications in accessibility and assistive communication technologies.