Multi-modal emotion recognition for determining employee satisfaction

Abstract

Emotion Detection has been very popular in the field of research for a couple of years. In the past, emotion recognition has been studied and applied in order to detect the overall emotional state of a person using individual modalities such as facial recognition from images. However, in order to ensure the authenticity of the real time emotional state detected from the data that is received, it is required to use multiple modes. In our research, we have classified emotional states into six specific entities which are: Happiness, Sadness, Neutral, Disgust, Anger and Surprise. The real time emotional state of the candidate is classified into one of these entities according to the candidate’s response. We have used two important modes to detect the real time emotion of the candidate, Emotion recognition from facial expression as well as emotion recognition from sentimental analysis. For our research, we have mainly used the Convolutional Neural Network(CNN). We have trained and tested both the facial recognition and sentimental analysis datasets with all the six entities. Therefore, results can be obtained from both the modes in order to justify the candidate’s emotional state in real time. The two separate results from the individual independent artificial neural networks are then fed into a machine learning algorithm called Support Vector Machine (SVM) so that the final emotional state can be achieved. Our goal is to apply this multimodal emotion detection technique on employees in various offices and workplaces by asking them questions regarding their work so that their genuine emotions can be obtained from their answers. This is important because in this way, employee satisfaction in workplaces can be recognized which is vital for mental health as well as productivity. In fact, the mental health of the employees not only affects their individual well-being but it affects the overall productivity and environment of the workplace. In order to improve certain aspects of a workplace for better performance along with employee satisfaction and productivity, determining the emotional condition of each employee is vital