Towards accurate autism detection: a multimodal machine learning approach

Abstract

Autism Spectrum Disorder (ASD) is known as a complex neurodevelopmental condition characterized by a wide range of behavioral and communicative deficits. The detection of ASD has been established as a critical area of research, wherein automated and reliable approaches have been enabled through deep learning methodologies. In this study, a multimodal fusion framework was proposed, in which data from image, audio, video and text modalities was integrated to capture complementary behavioral cues. Specialized deep learning architectures were employed for each modality so that high-level feature representations could be extracted, which were subsequently combined at the decision-making stage to enhance classification robustness. The curated dataset, comprising individuals with and without autism, was used to assess the proposed multimodal model. An exceptional average accuracy of 98.61% and an F1-score of 98.61% were obtained across 5-fold cross-validation, thereby demonstrating the superior efficacy of integrating heterogeneous data sources compared to single-modality approaches. Interpretability methods, including SHAP, were utilized to elucidate the decisionmaking process, highlighting clinically relevant features such as gaze direction, vocal pitch and semantic content. Through its transparency, robustness and state-of-the-art performance, the proposed model is highlighted as a promising advancement toward practical, AI-driven, non-invasive autism screening tools.