Analyzing emotion patterns in gaming using CNNs on facial and vocal features

Abstract

People of different ages frequently use technology for leisure activities, and gaming is a common pastime for many. However, playing video games may cause significant changes in behavior, both positive and negative. Research into those changes has been ongoing for a long time. Most of the research was conducted using sophisticated medical environments. We propose a multimodal CNN-based approach to observe emotional changes in players using facial and speech cues extracted from gameplay video frames. A vast number of YouTube videos were collected from different online gaming streamers, and then image and audio datasets comprising hundreds of those videos were created. Utilizing Facial Expression Recognition (FER) and Speech Emotion Recognition (SER) methodologies, our objective was to identify patterns of behavioral changes during gaming sessions and longitudinally. Multiple models were employed for both SER and FER. For FER, DenseNet121 was fine-tuned and achieved the best performance, and for SER, a custom CNN architecture, specifically optimized for speech features like MFCCs and spectrograms, was developed and outperformed other SER models. In our research, we established the effectiveness of our approach in discerning patterns associated with behavioral changes.