Generative AI meets responsible AI and affective computing

Abstract

Generative AI, Responsible AI, and Affective Computing are transforming the future of artificial intelligence. The intersection of these fields represents a revolutionary breakthrough in computational technology. This thesis integrates these domains to develop a formalism for multidimensional emotional communication. By analysing image, voice, and text data, we address the challenge of detecting and generating emotions in real time, considering users’ gestures and interactions. We adopt an integrated approach based on deep neural network models across multiple modalities: text sentiment analysis, audio emotion detection, and facial expression recognition. In particular, we built our proposed approach using transformer-based models, including DistilRoBERTa, fine-tuned Wav2Vec2 on custom dataset, and DeepFace to process text, audio, and facial expression respectively. These pretrained models are trained for emotion classification with 6.7 million, 95 million, and 120 million trainable parameters, respectively. Natural Language Processing (NLP) models are used to interpret meanings and sentiments in text, while audio and image-based models detect emotional cues. The system adapts dynamically based on user feedback and incorporates Responsible AI practices such as bias detection, ethical safeguards, and safe interactions to ensure fairness and trustworthiness. Through practical experimentation and evaluation, we demonstrate that it is possible to build Generative AI systems capable of not only perceiving and reacting to human emotions but also generating emotionally appropriate responses. Potential applications include virtual assistants, mental health support tools, interactive storytelling systems, and educational platforms where enhanced emotional intelligence can significantly improve user experience.