A multimodal approach to dementia detection using contrastive learning and LLM–VLM assisted reasoning with guided prompting

Abstract

Early detection of dementia remains challenging due to the cost, limited accessibility and subjectivity of traditional clinical assessments. This study proposes a non-invasive multimodal dementia detection framework using spontaneous speech from the Pitt Corpus of DementiaBank by exploring contrastive learning using text–audio representation and reasoning-based foundation models, Large Language Models and Vision Language Models. Linguistic features capturing lexical diversity, syntactic complexity, coherence,etc were combined with acoustic features including pitch, jitter, shimmer, MFCC,etc with Contrastive Language Audio Pretraining (CLAP) based text and audio embeddings. Among classical machine learning classifiers, the best performing configuration using Random Forest with text, audio and handcrafted features achieved an accuracy of 90.83%, F1-score of 0.9083 and AUC of 0.9478, while LightGBM achieved 89.91% accuracy, 0.8989 F1-score and the highest AUC of 0.9675, demonstrating the effectiveness of multimodal fusion of text, audio and features over unimodal baselines. In addition, large language models were evaluated under instruction based inference without fine tuning, where GPT-OSS achieved the highest accuracy of 68.07% among the tested LLMs, outperforming Qwen3-4B and Mistral. Vision language models were further examined using different prompting techniques and the hybrid VLM + LLM reasoning pipeline consistently outperformed standalone VLM configurations, indicating that hierarchical reasoning enhances multimodal dementia classification. Overall, the findings show that contrastive multimodal learning achieves strong classification performance, while reasoning based LLM and VLM frameworks enhance interpretability highlighting the potential of AI assisted methods for early dementia screening.