A transfer learning framework for cross-script visual speech recognition

Abstract

Visual Speech Recognition (VSR), or commonly known as lip reading is a century old technique to retrieve speech information from visual cues. But this is still underexplored in the digital domain. The processing and linguistic complexities behind this technology has forced its capability to be limited in a very small scope, especially in cases like developing VSR model for languages originating from a different script which sounds and reads differently and there is no custom model built for that language or script. The research proposes a novel approach of cross-script VSR through the development of a transfer learning framework, tailored for languages that have data scarcity. Traditional VSR systems often struggle with language dependency and lack adaptability when applied across diverse scripts. In response, the research employs a systematic transfer learning strategy that leverages pre-trained models from already trained languages to enhance recognition capabilities in data-constrained environments. The research develops a Visual Efficient Conformerbased architecture and fine-tunes its performance on the LipBengal and Lip Reading in the Wild - Arabic (LRW-AR) datasets. By leveraging transfer learning technique, the research proposes a model that achieves 79.77% accuracy on LRW-AR with 20.23% Word Error Rate (WER) and 45.83% accuracy on LipBengal with 54.19% WER. The research then demonstrates an accuracy of 94.64% in LRW-AR and 74.73% in LipBengal when evaluated with top-10 predictions in the same settings. Furthermore, the research analyzes the impact of various transfer learning strategies, including encoder freezing, learning rate adjustments, data preprocessing, and augmentation techniques. The research proposes a model that pushes the development of inclusive communication technologies, significantly benefiting the hearing-impaired community and facilitating seamless cross-lingual interaction.