A viseme recognition system using lip curvature and neural networks to detect Bangla vowels

Abstract

Automatic Speech Recognition plays an important role in human-computer interaction, which can be applied in various vital applications like crime-fighting and helping the hearing-impaired. It consists of two domains – Audio Speech Recognition and Visual Speech Recognition. This thesis is based on Recognition of Speech in the visual domain only, i.e. it involves recognizing speech without the presence or support of any auditory signal. So far, a lot of research has been done on lip-reading in English and some amount on French and Chinese, as well as few other languages, but not much research has been done on lip-reading in Bengali. This thesis work provides a new approach to lip reading Bengali vowels using a combination of the curvature of the inner and outer lips and Neural Networks. The method uses a more robust and faster algorithm to detect the lip contour than conventional methods used so far, such as Active Contour Model, Active Appearance Model and Active Shape Models. The method used for feature extraction is also new. It makes use of coefficients of the curves of the inner and outer lips. This way, it makes use of a lesser number of parameters to represent the shape of the lip when pronouncing a vowel. Moreover, the method is also robust to alignment of lips at different angles and can work with low resolution pictures also. Finally, for recognition of the viseme, a Backpropagation Neural Network is trained and simulated using gradient descent method.