Generative AI meets responsible AI and affective computing

Abstract

Artificial Intelligence (AI) systems are increasingly expected to interpret and respond to human emotions in real time, yet most existing solutions remain limited to unimodal or ethically unregulated approaches. In this research, we propose a unified, multimodal emotion recognition (MER) framework that integrates Generative AI (GenAI) and Responsible AI (RAI) with Affective Computing to enhance human-computer emotional interactions. We explore real-time emotion recognition and response by analyzing image, speech, and text, while also incorporating user gestures and behavioral signals. We construct a custom dataset of 1000 annotated video samples spanning seven emotion classes. We built our proposed approach using transformer-based models, including DistilRoBERTa, fine-tuned Wav2Vec2, and DeepFace to process text for sentiment classification, audio for emotion detection, and facial image modalities for facial expression analysis. A majority-vote fusion strategy combines the outputs of these models to identify dominant emotional states and gradually learns from user feedback, adapting to behave more human-like and empathetic. Additionally, our system also incorporates RAI layers to ensure ethical safeguards through bias and threat detection. Finally, a FLAN-T5-based generative module produces natural language summaries that reflect both the emotional content and ethical assessments. Our proposed method achieves an overall accuracy of 81%, with F1-scores of 0.94 and 0.90 for anger and disgust, respectively. Our approach suggests the potential for significant improvements beyond unimodal baselines, enabling ethically aware and emotionally intelligent applications such as virtual assistants, mental health support systems, and emotion-adaptive learning platforms.