Introducing a Bangla sentence gloss pair dataset for Bangla sign language translation and research

Abstract

Bangla Sign Language translation and recognition has been an evolving research topic throughout the years. However, existing research on this field is limited to word and alphabet level detection. For a more continuous sentence level detection of spoken Bangla sentences and their corresponding signed gestures, structured and comprehensive annotations are essential. Therefore, in this paper we introduce a dataset that consists of Bangla sentences and their matching gloss sequence pairs. Gloss sequences are made up of individual glosses which are Bangla sign supported words and serve as an intermediate representation for a continuous sign. Our dataset consists of 1000 high quality Bangla sentences that are manually annotated into a gloss sequence by a professional signer. With no available open-source datasets on Bangla sentence and gloss pairs, we additionally augment our dataset with 3000 synthetic samples. For the augmentation process, we introduce a RAG-based pipeline which incorporates rule-based linguistic strategies and prompt engineering techniques that we have adopted by critically analyzing our human annotated sentencegloss pairs and by working closely with our professional signer. Furthermore, we finetune several transformer-based models such as mBart-50, Google mT5, GPT4.1- nano and perform BLEU score based evaluations to determine which model performs the best in the task of Sentence-to-gloss translation. Finally, based on these evaluation metrics we also compare how our dataset performs against the Phoenix-2014T dataset.