Lightweight Visual Question Answering (VQA) model for skin disease detection

Abstract

Visual Question Answering (VQA) is an area of artificial intelligence that combines image analysis with natural language understanding to generate context-aware answers to queries. Its application in the medical field, particularly in dermatology, holds significant importance by enabling accessible, interpretable, and efficient diagnostic support. Currently, there are plenty of good skin disease classification datasets, however, there is a lack of structured VQA datasets for skin diseases that can be used for training as well as benchmarking models. Hence, we developed a custom dataset of 1,038 images for 11 disease classes, with seven question-answer pairs per image. Moreover, existing VQA models are extremely heavy weight and require specialized hardware to train and run. To address this, our research proposes a lightweight VQA model pipeline capable of identifying common skin diseases from images and responding to clinically relevant questions related to disease name, severity, causes, diagnostic approach, prevention, contagiousness, and cancer risk. The model uses a modular architecture that integrates a Vision Transformer (ViT) with 86 million parameters for image encoding and MiniLM, a transformer-based text encoder with 22 million parameters. It achieved a high accuracy of 94.87% while minimizing computational requirements. In addition, we have also trained state-of-the-art vision language models such as Gemma-3, QwenVl-2.5, LLaVA-1.5, and BLIP-2 using our dataset for comparison. Among these, BLIP-2 achieved the highest Sentence-BERT score of 81.41%, indicating strong alignment between predicted and reference answers.