Developing an intelligent virtual assistant for extended reality environments: a multi-modal approach for understanding user intent and context using computer vision, natural language processing, and large language model

Abstract

Virtual reality (VR) and Mixed reality (MR) technologies have the capacity to enhance and revolutionize a variety of industry sectors, providing individuals with greater immersive and engaging interactions. Nevertheless, modern virtual reality technologies lack a virtual intellectual assistant that recognises and responds to users’ visual and auditory inputs. Thus hindering users from executing tasks, accessing information, and interacting with the overall virtual system environment. The goal of this research is to develop an immersive and intelligent virtual reality assistant to work with individual users’ contextual intent. The proposed Virtual Reality Intelligent Assistant (VRIA) will be able to understand and respond to users’ auditory and visual inputs and help users’ daily endeavours involving task automation, information retrieval, navigation, object recognition, entertainment, personalisation, education, home automation and communication to enhance overall productivity. VRIA offers the ability to leverage natural language processing, computer vision and large language models for understanding visual and auditory inputs. The virtual ecosystem enables user-friendly, tailored, personalised and highly adaptable VR environments to accomplish real-world endeavours. In addition with the incorporation of powerful UI, IoT devices, Mixed Reality attributes, and Cloud Computing infrastructure, the Virtual Reality Intelligent Assistant (VRIA) will meet the requirement for real virtual world interactions and be scalable and flexible across multiple VR/MR platforms and applications.