A multi-modal intelligent virtual assistant for extended reality environments based on computer vision, natural language processing, and large language models

Abstract

Extended Reality (XR) environments increasingly demand natural and adaptive interaction mechanisms beyond traditional menu-driven interfaces. This paper presents a multi-modal intelligent virtual assistant for XR that jointly understands user intent and contextual information by integrating computer vision, natural language processing (NLP), and large language models (LLMs). The key novelty of the proposed approach lies in a context-grounded reasoning framework that fuses real-time visual perception with conversational state to generate reliable and explainable assistant actions. Visual information, such as objects, spatial cues, and user activities, is transformed into a structured scene representation, which is combined with linguistic intent signals to support accurate decision-making in immersive environments. Unlike conventional virtual assistants that process visual and textual inputs independently, the proposed system employs confidence-aware multi-modal fusion and retrieval-augmented LLM reasoning to maintain contextual consistency and reduce ambiguous responses. The framework is designed in a modular and engine-agnostic manner, enabling seamless integration with common XR platforms for applications including object-aware guidance, interactive task support, and contextual information retrieval. Experimental evaluation is conducted using task-completion scenarios and user experience metrics, demonstrating improved interaction naturalness, reduced user effort, and higher contextual accuracy compared to unimodal and loosely coupled multimodal baselines. These results highlight the effectiveness of multi-modal grounding for developing intelligent, trustworthy, and user-centric virtual assistants in extended reality environments.